[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:59:02 dist-test-slave-zpvx krb5kdc[10726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:59:02 dist-test-slave-zpvx krb5kdc[10726](info): set up 2 sockets
Aug 12 02:59:02 dist-test-slave-zpvx krb5kdc[10726](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 02:59:04.090505 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.394s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 02:59:04 dist-test-slave-zpvx krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:59:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 02:59:04.305548 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41161
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:41161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:04.645673 10749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:04.646736 10749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:04.647665 10749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:04.689419 10749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:59:04.689895 10749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:04.690351 10749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:04.690687 10749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:59:04.690960 10749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:59:04.729872 10749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:41161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:04.730993 10749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:04.731982 10749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:04.751319 10758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:04.752100 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:04.754519 10768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:04.754099 10749 server_base.cc:1034] running on GCE node
I20260812 02:59:05.959115 10749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:05.962867 10749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:05.964349 10749 hybrid_clock.cc:648] HybridClock initialized: now 1786503545964317 us; error 111 us; skew 500 ppm
I20260812 02:59:05.965206 10749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:05 dist-test-slave-zpvx krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503545, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:05.971877 10749 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 02:59:05.978652 10749 webserver.cc:460] Webserver started at http://127.10.118.126:33059/ using document root <none> and password file <none>
I20260812 02:59:05.979591 10749 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:05.979836 10749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:05.980365 10749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:05.985929 10749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "04f69793966e449ab1591990ba2d180e"
format_stamp: "Formatted at 2026-08-12 02:59:05 on dist-test-slave-zpvx"
server_key: "daadbe12ae1b19b061e8bc273c66aa9f"
server_key_iv: "9ccb4de845eea258b5517d44fb269429"
server_key_version: "encryptionkey@0"
I20260812 02:59:05.987218 10749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "04f69793966e449ab1591990ba2d180e"
format_stamp: "Formatted at 2026-08-12 02:59:05 on dist-test-slave-zpvx"
server_key: "daadbe12ae1b19b061e8bc273c66aa9f"
server_key_iv: "9ccb4de845eea258b5517d44fb269429"
server_key_version: "encryptionkey@0"
I20260812 02:59:05.996483 10749 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s
I20260812 02:59:06.003036 10789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:06.004063 10749 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:59:06.004480 10749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "04f69793966e449ab1591990ba2d180e"
format_stamp: "Formatted at 2026-08-12 02:59:05 on dist-test-slave-zpvx"
server_key: "daadbe12ae1b19b061e8bc273c66aa9f"
server_key_iv: "9ccb4de845eea258b5517d44fb269429"
server_key_version: "encryptionkey@0"
I20260812 02:59:06.004870 10749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:06.041828 10749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:06.047925 10749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:06.048439 10749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:06.135807 10749 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:41161
I20260812 02:59:06.136015 10876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:41161 every 8 connection(s)
I20260812 02:59:06.138729 10749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 02:59:06.144302 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 10749
I20260812 02:59:06.144758 10878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:06.144821 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 02:59:06.147248 10713 external_mini_cluster.cc:1428] Setting key f08794388431339a4bc2960d164c80b5
I20260812 02:59:06.165799 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e: Bootstrap starting.
I20260812 02:59:06.171978 10878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:06.173785 10878 log.cc:824] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:06.179796 10878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e: No bootstrap required, opened a new log
Aug 12 02:59:06 dist-test-slave-zpvx krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:06.209897 10878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f69793966e449ab1591990ba2d180e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41161 } }
I20260812 02:59:06.210881 10878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:06.211272 10878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04f69793966e449ab1591990ba2d180e, State: Initialized, Role: FOLLOWER
I20260812 02:59:06.212296 10878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f69793966e449ab1591990ba2d180e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41161 } }
I20260812 02:59:06.213054 10878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:59:06.213433 10878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:59:06.213830 10878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:06.220050 10878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f69793966e449ab1591990ba2d180e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41161 } }
I20260812 02:59:06.221010 10878 leader_election.cc:302] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04f69793966e449ab1591990ba2d180e; no voters:
I20260812 02:59:06.222798 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:06.150985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.10.118.126:41161)
0812 02:59:06.154131 (+ 3146us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:06.154181 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:06.166009 (+ 11828us) server_negotiation.cc:407] Connection header received
0812 02:59:06.187023 (+ 21014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:06.187079 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:06.187636 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:06.188088 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:06.192530 (+ 4442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:06.195647 (+ 3117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:06.198132 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:06.199525 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:06.199877 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:06.207368 (+ 7491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:06.207417 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:06.207468 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:06.207634 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:06.213672 (+ 6038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:06.215102 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:06.215139 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:06.215168 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:06.215485 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:06.216270 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:06.216301 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:06.216327 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:06.217465 (+ 1138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:06.217935 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:06.218845 (+ 910us) server_negotiation.cc:299] Negotiation successful
0812 02:59:06.219500 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2540,"thread_start_us":2010,"threads_started":1}
I20260812 02:59:06.223760 10878 leader_election.cc:288] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:59:06.224591 10885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:06.229208 10885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [term 1 LEADER]: Becoming Leader. State: Replica: 04f69793966e449ab1591990ba2d180e, State: Running, Role: LEADER
I20260812 02:59:06.230091 10878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:59:06.230000 10885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f69793966e449ab1591990ba2d180e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41161 } }
I20260812 02:59:06.245455 10893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04f69793966e449ab1591990ba2d180e. Latest consensus state: current_term: 1 leader_uuid: "04f69793966e449ab1591990ba2d180e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f69793966e449ab1591990ba2d180e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41161 } } }
I20260812 02:59:06.245357 10888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04f69793966e449ab1591990ba2d180e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04f69793966e449ab1591990ba2d180e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41161 } } }
I20260812 02:59:06.246613 10893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:06.246685 10888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:06.253064 10906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:59:06.264883 10906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:59:06.289366 10906 catalog_manager.cc:1347] Generated new cluster ID: ef1a36c0193b492caf29dc6645474241
I20260812 02:59:06.289647 10906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:59:06.314143 10906 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:59:06.316910 10906 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:59:06.340399 10906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04f69793966e449ab1591990ba2d180e: Generated new TSK 0
I20260812 02:59:06.341614 10906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 02:59:06.412328 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41161
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:06.757965 10932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:06.758590 10932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:06.759303 10932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:06.790886 10932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:06.791285 10932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:06.792354 10932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 02:59:06.828069 10932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:06.829291 10932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:06.830389 10932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:06.855378 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:06.856846 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:06.860482 10932 server_base.cc:1034] running on GCE node
W20260812 02:59:06.857636 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:08.111258 10932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:08.113662 10932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:08.115067 10932 hybrid_clock.cc:648] HybridClock initialized: now 1786503548115034 us; error 56 us; skew 500 ppm
I20260812 02:59:08.115931 10932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:08 dist-test-slave-zpvx krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503548, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:08.127496 10932 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 02:59:08.133608 10932 webserver.cc:460] Webserver started at http://127.10.118.65:36347/ using document root <none> and password file <none>
I20260812 02:59:08.134805 10932 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:08.135054 10932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:08.135619 10932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:08.143126 10932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "865f7331ac0f4497b1442024bed4e257"
format_stamp: "Formatted at 2026-08-12 02:59:08 on dist-test-slave-zpvx"
server_key: "38ef6615081b861e04f3154ad77fda9f"
server_key_iv: "9d5710ea0147400cdb86e39513b53985"
server_key_version: "encryptionkey@0"
I20260812 02:59:08.144690 10932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "865f7331ac0f4497b1442024bed4e257"
format_stamp: "Formatted at 2026-08-12 02:59:08 on dist-test-slave-zpvx"
server_key: "38ef6615081b861e04f3154ad77fda9f"
server_key_iv: "9d5710ea0147400cdb86e39513b53985"
server_key_version: "encryptionkey@0"
I20260812 02:59:08.153000 10932 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 02:59:08.158986 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:08.160012 10932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20260812 02:59:08.160333 10932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "865f7331ac0f4497b1442024bed4e257"
format_stamp: "Formatted at 2026-08-12 02:59:08 on dist-test-slave-zpvx"
server_key: "38ef6615081b861e04f3154ad77fda9f"
server_key_iv: "9d5710ea0147400cdb86e39513b53985"
server_key_version: "encryptionkey@0"
I20260812 02:59:08.160734 10932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:08.204780 10932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:08.210677 10932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:08.211174 10932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:08.214264 10932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:08.219031 10932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:08.219230 10932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:08.219553 10932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:08.219697 10932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:08.401705 10932 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:45207
I20260812 02:59:08.401844 11176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:45207 every 8 connection(s)
I20260812 02:59:08.404464 10932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 02:59:08.410890 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 10932
I20260812 02:59:08.411336 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 02:59:08.412475 10713 external_mini_cluster.cc:1428] Setting key 12c54c3f2231ac342ed93f60fd55f0b5
Aug 12 02:59:08 dist-test-slave-zpvx krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503548, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 02:59:08.447842 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:08.409441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41161 (local address 127.10.118.65:33091)
0812 02:59:08.413266 (+ 3825us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:08.413337 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:08.418684 (+ 5347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:08.419817 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:08.419840 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:08.420818 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:08.421977 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:08.422021 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:08.424547 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:08.424583 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:08.427612 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:08.427673 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:08.428980 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:08.429026 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:08.429750 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:08.430982 (+ 1232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:08.431093 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:08.435422 (+ 4329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:08.441149 (+ 5727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:08.441179 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:08.441202 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:08.441852 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:08.442688 (+ 836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:08.442708 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:08.442725 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:08.442985 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:08.444346 (+ 1361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:08.444435 (+ 89us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:08.444879 (+ 444us) client_negotiation.cc:769] Sending connection context
0812 02:59:08.445202 (+ 323us) client_negotiation.cc:240] Negotiation successful
0812 02:59:08.445773 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3171,"thread_start_us":2267,"threads_started":1}
I20260812 02:59:08.449087 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:08.409008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:33091 (local address 127.10.118.126:41161)
0812 02:59:08.410243 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:08.410259 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:08.413512 (+ 3253us) server_negotiation.cc:407] Connection header received
0812 02:59:08.419043 (+ 5531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:08.419079 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:08.419367 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:08.419586 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:08.422300 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:08.423960 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:08.427888 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:08.428621 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:08.428840 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:08.435866 (+ 7026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:08.435909 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:08.435948 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:08.436121 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:08.440914 (+ 4793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:08.442113 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:08.442179 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:08.442212 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:08.442466 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:08.443232 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:08.443270 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:08.443298 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:08.444045 (+ 747us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:08.444321 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:08.445585 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0812 02:59:08.446181 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":589,"threads_started":1}
I20260812 02:59:08.451150 11179 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41161
I20260812 02:59:08.451577 11179 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:08.452606 11179 heartbeater.cc:511] Master 127.10.118.126:41161 requested a full tablet report, sending...
I20260812 02:59:08.455693 10815 ts_manager.cc:194] Registered new tserver with Master: 865f7331ac0f4497b1442024bed4e257 (127.10.118.65:45207)
I20260812 02:59:08.459328 10815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:33091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 02:59:08.470378 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41161
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:08.804118 11193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:08.804699 11193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:08.805429 11193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:08.836295 11193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:08.836694 11193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:08.837694 11193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 02:59:08.872453 11193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:08.873663 11193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:08.874718 11193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:08.895032 11208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:09.463922 11179 heartbeater.cc:503] Master 127.10.118.126:41161 was elected leader, sending a full tablet report...
W20260812 02:59:08.895105 11211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:08.895619 11213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:10.088532 11212 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 02:59:10.088675 11193 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 02:59:10.092294 11193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:10.095196 11193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:10.096638 11193 hybrid_clock.cc:648] HybridClock initialized: now 1786503550096604 us; error 53 us; skew 500 ppm
I20260812 02:59:10.097471 11193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:10 dist-test-slave-zpvx krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503550, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:10.103690 11193 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 02:59:10.109562 11193 webserver.cc:460] Webserver started at http://127.10.118.66:46621/ using document root <none> and password file <none>
I20260812 02:59:10.110435 11193 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:10.110615 11193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:10.111012 11193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:10.116279 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "3a556a4a15e54b8db40c74d350d47c6e"
format_stamp: "Formatted at 2026-08-12 02:59:10 on dist-test-slave-zpvx"
server_key: "b85dddbcaa3571dadbd61e1d3607c1b0"
server_key_iv: "f76c444fb618985e08e89c3f9ee61fb7"
server_key_version: "encryptionkey@0"
I20260812 02:59:10.117501 11193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "3a556a4a15e54b8db40c74d350d47c6e"
format_stamp: "Formatted at 2026-08-12 02:59:10 on dist-test-slave-zpvx"
server_key: "b85dddbcaa3571dadbd61e1d3607c1b0"
server_key_iv: "f76c444fb618985e08e89c3f9ee61fb7"
server_key_version: "encryptionkey@0"
I20260812 02:59:10.125465 11193 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 02:59:10.131280 11227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:10.132339 11193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 02:59:10.132773 11193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "3a556a4a15e54b8db40c74d350d47c6e"
format_stamp: "Formatted at 2026-08-12 02:59:10 on dist-test-slave-zpvx"
server_key: "b85dddbcaa3571dadbd61e1d3607c1b0"
server_key_iv: "f76c444fb618985e08e89c3f9ee61fb7"
server_key_version: "encryptionkey@0"
I20260812 02:59:10.133227 11193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:10.185930 11193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:10.191555 11193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:10.192016 11193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:10.194752 11193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:10.199095 11193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:10.199311 11193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:10.199504 11193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:10.199632 11193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:10.380316 11193 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:34499
I20260812 02:59:10.380456 11426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:34499 every 8 connection(s)
I20260812 02:59:10.383178 11193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 02:59:10.393721 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 11193
I20260812 02:59:10.394151 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 02:59:10.395097 10713 external_mini_cluster.cc:1428] Setting key 9277f796801f5bf0f1fc34371c2deb9a
Aug 12 02:59:10 dist-test-slave-zpvx krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503550, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:10.423206 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:10.388878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41161 (local address 127.10.118.66:51335)
0812 02:59:10.391692 (+ 2814us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:10.391763 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:10.397011 (+ 5248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:10.398151 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:10.398211 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:10.399388 (+ 1177us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:10.400269 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:10.400310 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:10.401766 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:10.401788 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:10.403483 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:10.403520 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:10.404743 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:10.404780 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:10.405242 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:10.406205 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:10.406271 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:10.409752 (+ 3481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:10.416357 (+ 6605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:10.416441 (+ 84us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:10.416477 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:10.417158 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:10.417948 (+ 790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:10.417967 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:10.417981 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:10.418245 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:10.419711 (+ 1466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:10.419740 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:10.420166 (+ 426us) client_negotiation.cc:769] Sending connection context
0812 02:59:10.420570 (+ 404us) client_negotiation.cc:240] Negotiation successful
0812 02:59:10.421102 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2247,"thread_start_us":1424,"threads_started":1}
I20260812 02:59:10.424289 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:10.388030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:51335 (local address 127.10.118.126:41161)
0812 02:59:10.389444 (+ 1414us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:10.389459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:10.391922 (+ 2463us) server_negotiation.cc:407] Connection header received
0812 02:59:10.397373 (+ 5451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:10.397396 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:10.397559 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:10.397808 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:10.400593 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:10.401500 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:10.403743 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:10.404347 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:10.404682 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:10.410225 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:10.410264 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:10.410297 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:10.410461 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:10.416123 (+ 5662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:10.417392 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:10.417440 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:10.417478 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:10.417750 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:10.418477 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:10.418514 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:10.418543 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:10.419436 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:10.419720 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:10.420870 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0812 02:59:10.421514 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":743,"threads_started":1}
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
I20260812 02:59:10.426338 11429 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41161
I20260812 02:59:10.426761 11429 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 02:59:10.427881 11429 heartbeater.cc:511] Master 127.10.118.126:41161 requested a full tablet report, sending...
I20260812 02:59:10.430222 10815 ts_manager.cc:194] Registered new tserver with Master: 3a556a4a15e54b8db40c74d350d47c6e (127.10.118.66:34499)
I20260812 02:59:10.431660 10815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:51335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 02:59:10.453334 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41161
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:10.799736 11442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:10.800233 11442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:10.800935 11442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:10.831984 11442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:10.832309 11442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:10.833349 11442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 02:59:10.868152 11442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:10.869500 11442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:10.870463 11442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:10.890730 11456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:11.435163 11429 heartbeater.cc:503] Master 127.10.118.126:41161 was elected leader, sending a full tablet report...
W20260812 02:59:10.890918 11454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:10.892493 11460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:10.894125 11442 server_base.cc:1034] running on GCE node
I20260812 02:59:12.083307 11442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:12.086483 11442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:12.087965 11442 hybrid_clock.cc:648] HybridClock initialized: now 1786503552087934 us; error 49 us; skew 500 ppm
I20260812 02:59:12.088923 11442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503552, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:12.095372 11442 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 02:59:12.101609 11442 webserver.cc:460] Webserver started at http://127.10.118.67:36061/ using document root <none> and password file <none>
I20260812 02:59:12.102581 11442 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:12.102823 11442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:12.103303 11442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:12.108776 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "541b790c784140be965ddced462e8fca"
format_stamp: "Formatted at 2026-08-12 02:59:12 on dist-test-slave-zpvx"
server_key: "fcd472b8720e6aee5450cd24a38bd56d"
server_key_iv: "2e4803c7180b6f2a67e63069759b8ed7"
server_key_version: "encryptionkey@0"
I20260812 02:59:12.109984 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "541b790c784140be965ddced462e8fca"
format_stamp: "Formatted at 2026-08-12 02:59:12 on dist-test-slave-zpvx"
server_key: "fcd472b8720e6aee5450cd24a38bd56d"
server_key_iv: "2e4803c7180b6f2a67e63069759b8ed7"
server_key_version: "encryptionkey@0"
I20260812 02:59:12.118266 11442 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260812 02:59:12.124585 11480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:12.125680 11442 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:59:12.126034 11442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "541b790c784140be965ddced462e8fca"
format_stamp: "Formatted at 2026-08-12 02:59:12 on dist-test-slave-zpvx"
server_key: "fcd472b8720e6aee5450cd24a38bd56d"
server_key_iv: "2e4803c7180b6f2a67e63069759b8ed7"
server_key_version: "encryptionkey@0"
I20260812 02:59:12.126368 11442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:12.170459 11442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:12.175931 11442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:12.176323 11442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:12.179284 11442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:12.184017 11442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:12.184203 11442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:12.184471 11442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:12.184618 11442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:12.372409 11442 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:37795
I20260812 02:59:12.372670 11696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:37795 every 8 connection(s)
I20260812 02:59:12.375051 11442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 02:59:12.375852 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 11442
I20260812 02:59:12.376350 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 02:59:12.377456 10713 external_mini_cluster.cc:1428] Setting key d6fe5892582440c47e7ae70e89a1ff47
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503552, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:12.422659 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:12.381399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:36679 (local address 127.10.118.126:41161)
0812 02:59:12.390990 (+ 9591us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:12.391015 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:12.391069 (+ 54us) server_negotiation.cc:407] Connection header received
0812 02:59:12.396568 (+ 5499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:12.396604 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:12.396914 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:12.397126 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:12.400958 (+ 3832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.402707 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.405071 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.406229 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.406589 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:12.411807 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:12.411839 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:12.411864 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:12.411994 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:12.416824 (+ 4830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:12.417947 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:12.417969 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:12.417990 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:12.418248 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:12.418910 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:12.418928 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:12.418941 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:12.419426 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:12.419601 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:12.420638 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0812 02:59:12.420974 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9087,"thread_start_us":8790,"threads_started":1}
I20260812 02:59:12.423420 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:12.381754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41161 (local address 127.10.118.67:36679)
0812 02:59:12.390686 (+ 8932us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:12.390841 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:12.396147 (+ 5306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:12.397373 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:12.397398 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:12.398850 (+ 1452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:12.400464 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:12.400531 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.403076 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.403097 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:12.404765 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:12.404804 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.406614 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.406640 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:12.407037 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:12.407981 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:12.408050 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:12.411437 (+ 3387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:12.416997 (+ 5560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:12.417021 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:12.417046 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:12.417765 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:12.418420 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:12.418440 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:12.418460 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:12.418734 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:12.419595 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:12.419624 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:12.420053 (+ 429us) client_negotiation.cc:769] Sending connection context
0812 02:59:12.420484 (+ 431us) client_negotiation.cc:240] Negotiation successful
0812 02:59:12.421094 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6939,"thread_start_us":6007,"threads_started":1}
I20260812 02:59:12.425196 11699 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41161
I20260812 02:59:12.425571 11699 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:12.426739 11699 heartbeater.cc:511] Master 127.10.118.126:41161 requested a full tablet report, sending...
I20260812 02:59:12.429462 10815 ts_manager.cc:194] Registered new tserver with Master: 541b790c784140be965ddced462e8fca (127.10.118.67:37795)
I20260812 02:59:12.431324 10815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:36679
I20260812 02:59:12.444042 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503552, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 02:59:12.518110 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:12.491388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.10.118.65:45207)
0812 02:59:12.493851 (+ 2463us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:12.493876 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:12.494436 (+ 560us) server_negotiation.cc:407] Connection header received
0812 02:59:12.494756 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:12.494776 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:12.495032 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:12.495214 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:12.498082 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.499396 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.501532 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.502383 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.502679 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:12.507562 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:12.507607 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:12.507634 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:12.507728 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:12.512174 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:12.513406 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:12.513431 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:12.513446 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:12.513686 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:12.514253 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:12.514273 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:12.514288 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:12.514974 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:12.515378 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:12.515891 (+ 513us) server_negotiation.cc:299] Negotiation successful
0812 02:59:12.516361 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2071,"thread_start_us":1781,"threads_started":1}
W20260812 02:59:12.519811 11009 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33852
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[10726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[10726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 02:59:12.585909 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:12.562217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.10.118.65:45207)
0812 02:59:12.562898 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:12.562935 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:12.564190 (+ 1255us) server_negotiation.cc:407] Connection header received
0812 02:59:12.564642 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:12.564666 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:12.564835 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:12.565002 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:12.567428 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.568293 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.570217 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:12.570843 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:12.571168 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:12.576481 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:12.576509 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:12.576532 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:12.576630 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:12.580238 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:12.581743 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:12.581772 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:12.581789 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:12.581992 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:12.582549 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:12.582570 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:12.582585 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:12.583143 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:12.583399 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:12.583936 (+ 537us) server_negotiation.cc:299] Negotiation successful
0812 02:59:12.584275 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 02:59:12.588335 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 10932
I20260812 02:59:12.626433 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 11193
I20260812 02:59:12.657389 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 11442
I20260812 02:59:12.695629 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 10749
2026-08-12T02:59:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (10109 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[11739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[11739](info): set up 2 sockets
Aug 12 02:59:12 dist-test-slave-zpvx krb5kdc[11739](info): commencing operation
krb5kdc: starting...
W20260812 02:59:14.063023 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.292s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 02:59:14 dist-test-slave-zpvx krb5kdc[11739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:59:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 02:59:14.262848 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:34621
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:34621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:14.564414 11767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:14.565042 11767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:14.565662 11767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:14.595741 11767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:59:14.596110 11767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:14.596450 11767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:14.596719 11767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:59:14.596974 11767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:59:14.632266 11767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:34621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:34621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:14.633468 11767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:14.634524 11767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:14.652735 11778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:14.653349 11780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:14.655192 11767 server_base.cc:1034] running on GCE node
W20260812 02:59:14.652897 11777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:15.854915 11767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:15.858089 11767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:15.859534 11767 hybrid_clock.cc:648] HybridClock initialized: now 1786503555859474 us; error 81 us; skew 500 ppm
I20260812 02:59:15.860875 11767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:15 dist-test-slave-zpvx krb5kdc[11739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503555, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:15.871404 11767 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 02:59:15.876825 11767 webserver.cc:460] Webserver started at http://127.10.118.126:36071/ using document root <none> and password file <none>
I20260812 02:59:15.877763 11767 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:15.878006 11767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:15.878479 11767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:15.883833 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "6243225e4fd0477383c787cbda73838e"
format_stamp: "Formatted at 2026-08-12 02:59:15 on dist-test-slave-zpvx"
server_key: "f183bdb0f49cb8727ba842117df729b5"
server_key_iv: "dc145d7ae8ef7128efcbb76a96651cfa"
server_key_version: "encryptionkey@0"
I20260812 02:59:15.885063 11767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "6243225e4fd0477383c787cbda73838e"
format_stamp: "Formatted at 2026-08-12 02:59:15 on dist-test-slave-zpvx"
server_key: "f183bdb0f49cb8727ba842117df729b5"
server_key_iv: "dc145d7ae8ef7128efcbb76a96651cfa"
server_key_version: "encryptionkey@0"
I20260812 02:59:15.892761 11767 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 02:59:15.898324 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:15.899359 11767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:59:15.899703 11767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "6243225e4fd0477383c787cbda73838e"
format_stamp: "Formatted at 2026-08-12 02:59:15 on dist-test-slave-zpvx"
server_key: "f183bdb0f49cb8727ba842117df729b5"
server_key_iv: "dc145d7ae8ef7128efcbb76a96651cfa"
server_key_version: "encryptionkey@0"
I20260812 02:59:15.900089 11767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:15.944128 11767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:15.949728 11767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:15.950222 11767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:16.019411 11767 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:34621
I20260812 02:59:16.019718 11878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:34621 every 8 connection(s)
I20260812 02:59:16.022341 11767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 02:59:16.027618 11880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:16.032680 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 11767
I20260812 02:59:16.033226 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 02:59:16.034781 10713 external_mini_cluster.cc:1428] Setting key dba9979adeb692585182683b57dd039f
I20260812 02:59:16.051046 11880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e: Bootstrap starting.
I20260812 02:59:16.056257 11880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:16.058179 11880 log.cc:824] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e: Log is configured to *not* fsync() on all Append() calls
Aug 12 02:59:16 dist-test-slave-zpvx krb5kdc[11739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:16.063632 11880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e: No bootstrap required, opened a new log
I20260812 02:59:16.070600 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:16.037735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.10.118.126:34621)
0812 02:59:16.040588 (+ 2853us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:16.040618 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:16.040671 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:59:16.044800 (+ 4129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:16.044847 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:16.045297 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:16.045754 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:16.048666 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:16.050482 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:16.052662 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:16.053599 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:16.053970 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:16.059730 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:16.059760 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:16.059792 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:16.059898 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:16.064128 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:16.065287 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:16.065309 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:16.065328 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:16.065563 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:16.066215 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:16.066235 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:16.066248 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:16.066913 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:16.067430 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:16.068133 (+ 703us) server_negotiation.cc:299] Negotiation successful
0812 02:59:16.068747 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2237,"thread_start_us":1761,"threads_started":1}
I20260812 02:59:16.084087 11880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6243225e4fd0477383c787cbda73838e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34621 } }
I20260812 02:59:16.084978 11880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:16.085238 11880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6243225e4fd0477383c787cbda73838e, State: Initialized, Role: FOLLOWER
I20260812 02:59:16.086028 11880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6243225e4fd0477383c787cbda73838e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34621 } }
I20260812 02:59:16.086562 11880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:59:16.086846 11880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:59:16.087180 11880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:16.091449 11880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6243225e4fd0477383c787cbda73838e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34621 } }
I20260812 02:59:16.092257 11880 leader_election.cc:302] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6243225e4fd0477383c787cbda73838e; no voters:
I20260812 02:59:16.093981 11880 leader_election.cc:288] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:59:16.094398 11888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:16.096984 11888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [term 1 LEADER]: Becoming Leader. State: Replica: 6243225e4fd0477383c787cbda73838e, State: Running, Role: LEADER
I20260812 02:59:16.097752 11888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6243225e4fd0477383c787cbda73838e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34621 } }
I20260812 02:59:16.098651 11880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:59:16.108487 11890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6243225e4fd0477383c787cbda73838e. Latest consensus state: current_term: 1 leader_uuid: "6243225e4fd0477383c787cbda73838e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6243225e4fd0477383c787cbda73838e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34621 } } }
I20260812 02:59:16.109387 11890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:16.111900 11889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6243225e4fd0477383c787cbda73838e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6243225e4fd0477383c787cbda73838e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34621 } } }
I20260812 02:59:16.112751 11889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:16.117681 11899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:59:16.129276 11899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:59:16.143765 11899 catalog_manager.cc:1347] Generated new cluster ID: 02751cf4337143d8b0ef7b3b7fb468df
I20260812 02:59:16.144048 11899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:59:16.162324 11899 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:59:16.164171 11899 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:59:16.177278 11899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6243225e4fd0477383c787cbda73838e: Generated new TSK 0
I20260812 02:59:16.178180 11899 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 02:59:16.236874 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34621
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:16.545583 11927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:16.546141 11927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:16.546842 11927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:16.578202 11927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:16.578574 11927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:16.579514 11927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 02:59:16.615355 11927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:16.616663 11927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:16.617705 11927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:16.638759 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:16.641113 11945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:16.638877 11938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:18.012665 11940 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1372 milliseconds
I20260812 02:59:18.012782 11927 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 02:59:18.014082 11927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:18.016817 11927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:18.018337 11927 hybrid_clock.cc:648] HybridClock initialized: now 1786503558018270 us; error 82 us; skew 500 ppm
I20260812 02:59:18.019380 11927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:18 dist-test-slave-zpvx krb5kdc[11739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503558, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:18.026952 11927 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 02:59:18.035720 11927 webserver.cc:460] Webserver started at http://127.10.118.65:43357/ using document root <none> and password file <none>
I20260812 02:59:18.036940 11927 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:18.037202 11927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:18.037741 11927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:18.045200 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "b7dc63955f0c46dba499bdcb4f7e2648"
format_stamp: "Formatted at 2026-08-12 02:59:18 on dist-test-slave-zpvx"
server_key: "3717edea51190eace79d17234f7dad9b"
server_key_iv: "53604574ffde679d659f7619dd97bbfd"
server_key_version: "encryptionkey@0"
I20260812 02:59:18.046690 11927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "b7dc63955f0c46dba499bdcb4f7e2648"
format_stamp: "Formatted at 2026-08-12 02:59:18 on dist-test-slave-zpvx"
server_key: "3717edea51190eace79d17234f7dad9b"
server_key_iv: "53604574ffde679d659f7619dd97bbfd"
server_key_version: "encryptionkey@0"
I20260812 02:59:18.055768 11927 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20260812 02:59:18.062968 11955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:18.064160 11927 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 02:59:18.064560 11927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "b7dc63955f0c46dba499bdcb4f7e2648"
format_stamp: "Formatted at 2026-08-12 02:59:18 on dist-test-slave-zpvx"
server_key: "3717edea51190eace79d17234f7dad9b"
server_key_iv: "53604574ffde679d659f7619dd97bbfd"
server_key_version: "encryptionkey@0"
I20260812 02:59:18.065035 11927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:18.138725 11927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:18.144675 11927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:18.145169 11927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:18.147845 11927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:18.152042 11927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:18.152283 11927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:18.152603 11927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:18.152784 11927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:18.324105 11927 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:39895
I20260812 02:59:18.324227 12176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:39895 every 8 connection(s)
I20260812 02:59:18.326857 11927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 02:59:18.335227 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 11927
I20260812 02:59:18.335646 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 02:59:18.336788 10713 external_mini_cluster.cc:1428] Setting key 1d3dc7c07b332486cdb73d09655787b1
Aug 12 02:59:18 dist-test-slave-zpvx krb5kdc[11739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503558, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 02:59:18.368409 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:18.331041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:57585 (local address 127.10.118.126:34621)
0812 02:59:18.332586 (+ 1545us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:18.332614 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:18.334251 (+ 1637us) server_negotiation.cc:407] Connection header received
0812 02:59:18.339371 (+ 5120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:18.339392 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:18.339560 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:18.339702 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:18.342244 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:18.344032 (+ 1788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:18.349341 (+ 5309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:18.349993 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:18.350204 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:18.356432 (+ 6228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:18.356474 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:18.356514 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:18.356672 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:18.362165 (+ 5493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:18.363389 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:18.363409 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:18.363423 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:18.363607 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:18.364412 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:18.364443 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:18.364463 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:18.365102 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:18.365288 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:18.366630 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0812 02:59:18.366962 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":657,"threads_started":1}
I20260812 02:59:18.370604 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:18.331432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34621 (local address 127.10.118.65:57585)
0812 02:59:18.334030 (+ 2598us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:18.334100 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:18.338996 (+ 4896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:18.340015 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:18.340040 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:18.341084 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:18.341950 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:18.341992 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:18.346823 (+ 4831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:18.346844 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:18.349019 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:18.349098 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:18.350335 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:18.350378 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:18.351022 (+ 644us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:18.352033 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:18.352138 (+ 105us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:18.356006 (+ 3868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:18.362354 (+ 6348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:18.362378 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:18.362404 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:18.363197 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:18.363825 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:18.363862 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:18.363892 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:18.364157 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:18.365318 (+ 1161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:18.365364 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:18.366012 (+ 648us) client_negotiation.cc:769] Sending connection context
0812 02:59:18.366469 (+ 457us) client_negotiation.cc:240] Negotiation successful
0812 02:59:18.367291 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2160,"thread_start_us":1426,"threads_started":1}
I20260812 02:59:18.372749 12177 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34621
I20260812 02:59:18.373148 12177 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:18.374070 12177 heartbeater.cc:511] Master 127.10.118.126:34621 requested a full tablet report, sending...
I20260812 02:59:18.376628 11820 ts_manager.cc:194] Registered new tserver with Master: b7dc63955f0c46dba499bdcb4f7e2648 (127.10.118.65:39895)
I20260812 02:59:18.378566 11820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:57585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 02:59:18.390952 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34621
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:18.701839 12188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:18.702376 12188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:18.703059 12188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:18.734092 12188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:18.734478 12188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:18.735466 12188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 02:59:18.771051 12188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:18.772282 12188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:18.773355 12188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:18.794864 12200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:19.382629 12177 heartbeater.cc:503] Master 127.10.118.126:34621 was elected leader, sending a full tablet report...
W20260812 02:59:18.796038 12197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:18.795439 12202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:18.796425 12188 server_base.cc:1034] running on GCE node
I20260812 02:59:19.996083 12188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:19.999022 12188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:20.000483 12188 hybrid_clock.cc:648] HybridClock initialized: now 1786503560000432 us; error 66 us; skew 500 ppm
I20260812 02:59:20.001309 12188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:20 dist-test-slave-zpvx krb5kdc[11739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:20.007126 12188 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 02:59:20.012315 12188 webserver.cc:460] Webserver started at http://127.10.118.66:45705/ using document root <none> and password file <none>
I20260812 02:59:20.013316 12188 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:20.013577 12188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:20.014142 12188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:20.019547 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "91892ed5e0184a95bd008ae9010be3dc"
format_stamp: "Formatted at 2026-08-12 02:59:20 on dist-test-slave-zpvx"
server_key: "d6b80ed82946cbe198d8d7be8e21d83e"
server_key_iv: "965cc7ea2693390b5a781153722444b3"
server_key_version: "encryptionkey@0"
I20260812 02:59:20.021205 12188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "91892ed5e0184a95bd008ae9010be3dc"
format_stamp: "Formatted at 2026-08-12 02:59:20 on dist-test-slave-zpvx"
server_key: "d6b80ed82946cbe198d8d7be8e21d83e"
server_key_iv: "965cc7ea2693390b5a781153722444b3"
server_key_version: "encryptionkey@0"
I20260812 02:59:20.029412 12188 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260812 02:59:20.035012 12215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:20.035992 12188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 02:59:20.036329 12188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "91892ed5e0184a95bd008ae9010be3dc"
format_stamp: "Formatted at 2026-08-12 02:59:20 on dist-test-slave-zpvx"
server_key: "d6b80ed82946cbe198d8d7be8e21d83e"
server_key_iv: "965cc7ea2693390b5a781153722444b3"
server_key_version: "encryptionkey@0"
I20260812 02:59:20.036772 12188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:20.077490 12188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:20.082762 12188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:20.083244 12188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:20.085808 12188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:20.089867 12188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:20.090142 12188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:20.090461 12188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:20.090612 12188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:20.224591 12188 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:43327
I20260812 02:59:20.224727 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:43327 every 8 connection(s)
I20260812 02:59:20.227123 12188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 02:59:20.231186 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 12188
I20260812 02:59:20.231689 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 02:59:20.233228 10713 external_mini_cluster.cc:1428] Setting key fc9224f2036ce1cbb2f2fd94a40bf214
Aug 12 02:59:20 dist-test-slave-zpvx krb5kdc[11739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503560, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 02:59:20.263607 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:20.231750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:59701 (local address 127.10.118.126:34621)
0812 02:59:20.233239 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:20.233264 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:20.234816 (+ 1552us) server_negotiation.cc:407] Connection header received
0812 02:59:20.239831 (+ 5015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:20.239876 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:20.240043 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:20.240231 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:20.242515 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:20.243434 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:20.245968 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:20.246583 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:20.246876 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:20.252043 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:20.252066 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:20.252084 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:20.252185 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:20.257384 (+ 5199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:20.258764 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:20.258785 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:20.258799 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:20.258995 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:20.259677 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:20.259696 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:20.259709 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:20.260222 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:20.260467 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:20.261822 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0812 02:59:20.262168 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":638,"threads_started":1}
I20260812 02:59:20.265713 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:20.231812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34621 (local address 127.10.118.66:59701)
0812 02:59:20.234529 (+ 2717us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:20.234611 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:20.239485 (+ 4874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:20.240533 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:20.240570 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:20.241370 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:20.242234 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:20.242277 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:20.243692 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:20.243712 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:20.245713 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:20.245751 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:20.246914 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:20.246949 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:20.247391 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:20.248302 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:20.248412 (+ 110us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:20.251707 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:20.257681 (+ 5974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:20.257707 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:20.257730 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:20.258560 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:20.259191 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:20.259217 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:20.259230 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:20.259509 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:20.260479 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:20.260549 (+ 70us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:20.261130 (+ 581us) client_negotiation.cc:769] Sending connection context
0812 02:59:20.261598 (+ 468us) client_negotiation.cc:240] Negotiation successful
0812 02:59:20.262410 (+ 812us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2289,"thread_start_us":1608,"threads_started":1}
I20260812 02:59:20.267891 12430 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34621
I20260812 02:59:20.268441 12430 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:20.269439 12430 heartbeater.cc:511] Master 127.10.118.126:34621 requested a full tablet report, sending...
I20260812 02:59:20.271780 11820 ts_manager.cc:194] Registered new tserver with Master: 91892ed5e0184a95bd008ae9010be3dc (127.10.118.66:43327)
I20260812 02:59:20.273279 11820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:59701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 02:59:20.289443 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34621
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:20.595607 12443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:20.596160 12443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:20.596947 12443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:20.628257 12443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:20.628677 12443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:20.629659 12443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 02:59:20.665499 12443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34621
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:20.666716 12443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:20.667797 12443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:20.688587 12453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:21.276880 12430 heartbeater.cc:503] Master 127.10.118.126:34621 was elected leader, sending a full tablet report...
W20260812 02:59:20.689704 12451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:20.688701 12449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:20.690624 12443 server_base.cc:1034] running on GCE node
I20260812 02:59:21.903637 12443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:21.906009 12443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:21.907415 12443 hybrid_clock.cc:648] HybridClock initialized: now 1786503561907405 us; error 73 us; skew 500 ppm
I20260812 02:59:21.908241 12443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:21 dist-test-slave-zpvx krb5kdc[11739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:21.920112 12443 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 02:59:21.925274 12443 webserver.cc:460] Webserver started at http://127.10.118.67:46393/ using document root <none> and password file <none>
I20260812 02:59:21.926190 12443 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:21.926378 12443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:21.926963 12443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:21.932143 12443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "b5cc776aab1747a88c2597cc290e6569"
format_stamp: "Formatted at 2026-08-12 02:59:21 on dist-test-slave-zpvx"
server_key: "eea6ce804d85302c57457472e9227440"
server_key_iv: "95d5b5d7bfff33836637e910f938495e"
server_key_version: "encryptionkey@0"
I20260812 02:59:21.933401 12443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "b5cc776aab1747a88c2597cc290e6569"
format_stamp: "Formatted at 2026-08-12 02:59:21 on dist-test-slave-zpvx"
server_key: "eea6ce804d85302c57457472e9227440"
server_key_iv: "95d5b5d7bfff33836637e910f938495e"
server_key_version: "encryptionkey@0"
I20260812 02:59:21.940613 12443 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 02:59:21.945945 12474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:21.946969 12443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 02:59:21.947278 12443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "b5cc776aab1747a88c2597cc290e6569"
format_stamp: "Formatted at 2026-08-12 02:59:21 on dist-test-slave-zpvx"
server_key: "eea6ce804d85302c57457472e9227440"
server_key_iv: "95d5b5d7bfff33836637e910f938495e"
server_key_version: "encryptionkey@0"
I20260812 02:59:21.947602 12443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:22.013072 12443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:22.018656 12443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:22.019101 12443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:22.022051 12443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:22.026295 12443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:22.026520 12443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:22.026789 12443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:22.026957 12443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:22.184855 12443 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:46803
I20260812 02:59:22.184969 12675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:46803 every 8 connection(s)
I20260812 02:59:22.187458 12443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 02:59:22.196007 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 12443
I20260812 02:59:22.196588 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 02:59:22.197978 10713 external_mini_cluster.cc:1428] Setting key c48ce4aa67af1a067d6f5e58c3085e6a
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[11739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503561, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:22.228910 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.192390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:49263 (local address 127.10.118.126:34621)
0812 02:59:22.194127 (+ 1737us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.194157 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.196717 (+ 2560us) server_negotiation.cc:407] Connection header received
0812 02:59:22.203360 (+ 6643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.203389 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.203588 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.203774 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:22.206533 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.207523 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.209995 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.210846 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.211189 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.216903 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:22.216954 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:22.216980 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:22.217165 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:22.221332 (+ 4167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.222949 (+ 1617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.222970 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.222987 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.223212 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.224127 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.224150 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.224164 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.224722 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:22.224908 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.226511 (+ 1603us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.226918 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":758,"threads_started":1}
I20260812 02:59:22.230449 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.192755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34621 (local address 127.10.118.67:49263)
0812 02:59:22.196398 (+ 3643us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:22.196499 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:22.202964 (+ 6465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:22.204065 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:22.204112 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:22.205103 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:22.206245 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.206298 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.207778 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.207799 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.209738 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.209778 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.211217 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.211254 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.211731 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.212663 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:22.212727 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:22.216387 (+ 3660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:22.221540 (+ 5153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:22.221571 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:22.221610 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:22.222734 (+ 1124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:22.223401 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:22.223434 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:22.223463 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:22.223844 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:22.224955 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:22.225008 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:22.225695 (+ 687us) client_negotiation.cc:769] Sending connection context
0812 02:59:22.226244 (+ 549us) client_negotiation.cc:240] Negotiation successful
0812 02:59:22.226994 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2872,"thread_start_us":2009,"threads_started":1}
I20260812 02:59:22.232759 12679 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34621
I20260812 02:59:22.233191 12679 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:22.234210 12679 heartbeater.cc:511] Master 127.10.118.126:34621 requested a full tablet report, sending...
I20260812 02:59:22.236812 11818 ts_manager.cc:194] Registered new tserver with Master: b5cc776aab1747a88c2597cc290e6569 (127.10.118.67:46803)
I20260812 02:59:22.238968 11818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:49263
I20260812 02:59:22.246031 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 02:59:22.284626 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.262345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.10.118.126:34621)
0812 02:59:22.262872 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.262907 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.264582 (+ 1675us) server_negotiation.cc:407] Connection header received
0812 02:59:22.264977 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.264997 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.265166 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.265339 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:22.267765 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.268740 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.270623 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.271363 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.271644 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.275088 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:22.275109 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:22.275125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:22.275262 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:22.279018 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.280057 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.280079 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.280092 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.280287 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.280927 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.280947 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.280963 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.281447 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:22.281967 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.282668 (+ 701us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.283160 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 02:59:22.293435 11818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:59:22.315321 11818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:22.371507 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.347369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.10.118.65:39895)
0812 02:59:22.350783 (+ 3414us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.350815 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.351536 (+ 721us) server_negotiation.cc:407] Connection header received
0812 02:59:22.351823 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.351860 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.352322 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.352690 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:22.356016 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.361398 (+ 5382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.365764 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.367373 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.367691 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.368178 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.369268 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.369642 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2838,"thread_start_us":6264,"threads_started":1}
I20260812 02:59:22.372292 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.347177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:39895 (local address 127.0.0.1:57352)
0812 02:59:22.350831 (+ 3654us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:22.350919 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:22.351273 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:22.353426 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:22.353455 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:22.353640 (+ 185us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:22.355472 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.355568 (+ 96us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.361724 (+ 6156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.361769 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.365411 (+ 3642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.365468 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.367812 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.367846 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.368444 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.368748 (+ 304us) client_negotiation.cc:769] Sending connection context
0812 02:59:22.368980 (+ 232us) client_negotiation.cc:240] Negotiation successful
0812 02:59:22.370177 (+ 1197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3113,"mutex_wait_us":506,"thread_start_us":2023,"threads_started":1}
I20260812 02:59:22.378223 12704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.349981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.10.118.66:43327)
0812 02:59:22.361508 (+ 11527us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.361548 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.361602 (+ 54us) server_negotiation.cc:407] Connection header received
0812 02:59:22.361775 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.361797 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.362033 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.362376 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:22.366181 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.368579 (+ 2398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.372857 (+ 4278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.374585 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.374912 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.375465 (+ 553us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.376077 (+ 612us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.376403 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11030,"thread_start_us":10714,"threads_started":1}
I20260812 02:59:22.377599 12057 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5bb2097884ebbac158e661337568d (DEFAULT_TABLE table=test-table [id=c7d44482152f42ff96ca1a9cd286affe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:22.379539 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5bb2097884ebbac158e661337568d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:22.381310 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.351319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:46803 (local address 127.0.0.1:40958)
0812 02:59:22.359982 (+ 8663us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:22.360046 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:22.360282 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:22.363556 (+ 3274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:22.363638 (+ 82us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:22.363776 (+ 138us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:22.365185 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.365243 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.368287 (+ 3044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.368317 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.374202 (+ 5885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.374240 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.377957 (+ 3717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.377988 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.378460 (+ 472us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.378625 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 02:59:22.378764 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 02:59:22.379061 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8019,"mutex_wait_us":2797,"thread_start_us":6821,"threads_started":1}
I20260812 02:59:22.381481 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.349696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:43327 (local address 127.0.0.1:54910)
0812 02:59:22.359094 (+ 9398us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:22.359387 (+ 293us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:22.359705 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:22.363211 (+ 3506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:22.363242 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:22.363349 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:22.365792 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.365845 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.368926 (+ 3081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.368971 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.372402 (+ 3431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.372473 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.375013 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.375064 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.375550 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.375731 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 02:59:22.375893 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 02:59:22.378966 (+ 3073us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8756,"mutex_wait_us":563,"thread_start_us":10145,"threads_started":1}
I20260812 02:59:22.382885 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.352494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.10.118.67:46803)
0812 02:59:22.362279 (+ 9785us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.362321 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.362379 (+ 58us) server_negotiation.cc:407] Connection header received
0812 02:59:22.362683 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.362730 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.363142 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.363758 (+ 616us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:22.365579 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.367988 (+ 2409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.374487 (+ 6499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.376891 (+ 2404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.377222 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.377701 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.380589 (+ 2888us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.380876 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9026,"thread_start_us":8546,"threads_started":1}
I20260812 02:59:22.386169 12302 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5bb2097884ebbac158e661337568d (DEFAULT_TABLE table=test-table [id=c7d44482152f42ff96ca1a9cd286affe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:22.387547 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5bb2097884ebbac158e661337568d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:22.387799 12566 tablet_service.cc:1467] Processing CreateTablet for tablet e2d5bb2097884ebbac158e661337568d (DEFAULT_TABLE table=test-table [id=c7d44482152f42ff96ca1a9cd286affe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:22.390209 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d5bb2097884ebbac158e661337568d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:22.412976 12708 tablet_bootstrap.cc:492] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648: Bootstrap starting.
I20260812 02:59:22.415753 12710 tablet_bootstrap.cc:492] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc: Bootstrap starting.
I20260812 02:59:22.422377 12708 tablet_bootstrap.cc:654] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:22.422988 12711 tablet_bootstrap.cc:492] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569: Bootstrap starting.
I20260812 02:59:22.424731 12710 tablet_bootstrap.cc:654] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:22.425102 12708 log.cc:824] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:22.430038 12710 log.cc:824] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:22.430758 12711 tablet_bootstrap.cc:654] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:22.432826 12708 tablet_bootstrap.cc:492] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648: No bootstrap required, opened a new log
I20260812 02:59:22.433531 12708 ts_tablet_manager.cc:1397] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20260812 02:59:22.433539 12711 log.cc:824] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:22.437122 12710 tablet_bootstrap.cc:492] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc: No bootstrap required, opened a new log
I20260812 02:59:22.437704 12710 ts_tablet_manager.cc:1397] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.004s
I20260812 02:59:22.439779 12711 tablet_bootstrap.cc:492] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569: No bootstrap required, opened a new log
I20260812 02:59:22.440245 12711 ts_tablet_manager.cc:1397] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260812 02:59:22.464257 12711 raft_consensus.cc:348] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.465107 12711 raft_consensus.cc:374] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:22.465451 12711 raft_consensus.cc:729] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cc776aab1747a88c2597cc290e6569, State: Initialized, Role: FOLLOWER
I20260812 02:59:22.466464 12711 consensus_queue.cc:260] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.467926 12708 raft_consensus.cc:348] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.468956 12708 raft_consensus.cc:374] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:22.469489 12708 raft_consensus.cc:729] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7dc63955f0c46dba499bdcb4f7e2648, State: Initialized, Role: FOLLOWER
I20260812 02:59:22.470604 12679 heartbeater.cc:503] Master 127.10.118.126:34621 was elected leader, sending a full tablet report...
I20260812 02:59:22.470585 12708 consensus_queue.cc:260] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.471236 12711 ts_tablet_manager.cc:1428] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569: Time spent starting tablet: real 0.031s user 0.026s sys 0.003s
I20260812 02:59:22.471940 12710 raft_consensus.cc:348] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.472684 12710 raft_consensus.cc:374] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:22.472934 12710 raft_consensus.cc:729] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91892ed5e0184a95bd008ae9010be3dc, State: Initialized, Role: FOLLOWER
I20260812 02:59:22.474256 12710 consensus_queue.cc:260] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.476054 12708 ts_tablet_manager.cc:1428] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648: Time spent starting tablet: real 0.042s user 0.036s sys 0.004s
I20260812 02:59:22.480523 12710 ts_tablet_manager.cc:1428] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc: Time spent starting tablet: real 0.042s user 0.033s sys 0.007s
W20260812 02:59:22.484243 12433 tablet.cc:2369] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:59:22.498456 12722 raft_consensus.cc:482] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:22.498914 12722 raft_consensus.cc:504] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.502036 12722 leader_election.cc:288] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7dc63955f0c46dba499bdcb4f7e2648 (127.10.118.65:39895), b5cc776aab1747a88c2597cc290e6569 (127.10.118.67:46803)
I20260812 02:59:22.516137 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.502497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:53947 (local address 127.10.118.65:39895)
0812 02:59:22.503184 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.503225 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.506052 (+ 2827us) server_negotiation.cc:407] Connection header received
0812 02:59:22.506476 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.506496 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.506679 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.506842 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:22.508127 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.509010 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.511819 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.512904 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.513337 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.513615 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.514235 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.514511 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260812 02:59:22.516502 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.503051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:34113 (local address 127.10.118.67:46803)
0812 02:59:22.503870 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.503908 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.506598 (+ 2690us) server_negotiation.cc:407] Connection header received
0812 02:59:22.506973 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.506992 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.507237 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.507480 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:22.508888 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.509717 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.512406 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.513414 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.513786 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.514054 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.514633 (+ 579us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.514915 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 02:59:22.517338 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.502571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:46803 (local address 127.10.118.66:34113)
0812 02:59:22.506428 (+ 3857us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:22.506466 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:22.506727 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:22.507544 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:22.507572 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:22.507670 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:22.508624 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.508662 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.509978 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.510000 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.512108 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.512144 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.513749 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.513789 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.514260 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.514420 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 02:59:22.514574 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 02:59:22.514878 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3397,"mutex_wait_us":78,"thread_start_us":3128,"threads_started":1}
I20260812 02:59:22.517333 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.502341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:39895 (local address 127.10.118.66:53947)
0812 02:59:22.505875 (+ 3534us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:22.505920 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:22.506208 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:22.506895 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:22.506922 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:22.507111 (+ 189us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:22.507868 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.507906 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.509297 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.509319 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.511541 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:22.511580 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.513366 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.513414 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:22.513853 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.513955 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 02:59:22.514166 (+ 211us) client_negotiation.cc:240] Negotiation successful
0812 02:59:22.514473 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3164,"mutex_wait_us":147,"thread_start_us":2864,"threads_started":1}
I20260812 02:59:22.519204 12599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5bb2097884ebbac158e661337568d" candidate_uuid: "91892ed5e0184a95bd008ae9010be3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cc776aab1747a88c2597cc290e6569" is_pre_election: true
I20260812 02:59:22.519204 12090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5bb2097884ebbac158e661337568d" candidate_uuid: "91892ed5e0184a95bd008ae9010be3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" is_pre_election: true
I20260812 02:59:22.520143 12090 raft_consensus.cc:2454] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91892ed5e0184a95bd008ae9010be3dc in term 0.
I20260812 02:59:22.520197 12599 raft_consensus.cc:2454] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91892ed5e0184a95bd008ae9010be3dc in term 0.
I20260812 02:59:22.521661 12217 leader_election.cc:302] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91892ed5e0184a95bd008ae9010be3dc, b5cc776aab1747a88c2597cc290e6569; no voters:
I20260812 02:59:22.522450 12722 raft_consensus.cc:2781] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:22.522876 12722 raft_consensus.cc:482] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:22.523157 12722 raft_consensus.cc:3037] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:22.528882 12722 raft_consensus.cc:504] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.530527 12722 leader_election.cc:288] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [CANDIDATE]: Term 1 election: Requested vote from peers b7dc63955f0c46dba499bdcb4f7e2648 (127.10.118.65:39895), b5cc776aab1747a88c2597cc290e6569 (127.10.118.67:46803)
I20260812 02:59:22.531343 12090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5bb2097884ebbac158e661337568d" candidate_uuid: "91892ed5e0184a95bd008ae9010be3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dc63955f0c46dba499bdcb4f7e2648"
I20260812 02:59:22.531589 12599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d5bb2097884ebbac158e661337568d" candidate_uuid: "91892ed5e0184a95bd008ae9010be3dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cc776aab1747a88c2597cc290e6569"
I20260812 02:59:22.531925 12090 raft_consensus.cc:3037] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:22.532089 12599 raft_consensus.cc:3037] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:22.537120 12090 raft_consensus.cc:2454] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91892ed5e0184a95bd008ae9010be3dc in term 1.
I20260812 02:59:22.537240 12599 raft_consensus.cc:2454] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91892ed5e0184a95bd008ae9010be3dc in term 1.
I20260812 02:59:22.538218 12222 leader_election.cc:302] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91892ed5e0184a95bd008ae9010be3dc, b7dc63955f0c46dba499bdcb4f7e2648; no voters:
I20260812 02:59:22.538846 12722 raft_consensus.cc:2781] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:22.540679 12722 raft_consensus.cc:686] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [term 1 LEADER]: Becoming Leader. State: Replica: 91892ed5e0184a95bd008ae9010be3dc, State: Running, Role: LEADER
I20260812 02:59:22.541725 12722 consensus_queue.cc:237] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } }
I20260812 02:59:22.554505 11817 catalog_manager.cc:5495] T e2d5bb2097884ebbac158e661337568d P 91892ed5e0184a95bd008ae9010be3dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 91892ed5e0184a95bd008ae9010be3dc (127.10.118.66). New cstate: current_term: 1 leader_uuid: "91892ed5e0184a95bd008ae9010be3dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7dc63955f0c46dba499bdcb4f7e2648" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 39895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91892ed5e0184a95bd008ae9010be3dc" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 43327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5cc776aab1747a88c2597cc290e6569" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 46803 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[11739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
W20260812 02:59:22.585804 12178 tablet.cc:2369] T e2d5bb2097884ebbac158e661337568d P b7dc63955f0c46dba499bdcb4f7e2648: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:59:22.593737 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.568237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.10.118.65:39895)
0812 02:59:22.568741 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.568762 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.570851 (+ 2089us) server_negotiation.cc:407] Connection header received
0812 02:59:22.571127 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.571145 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.571345 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.571488 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:22.574032 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.574988 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.577005 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.577568 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.577756 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.583397 (+ 5641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:22.583432 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:22.583464 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:22.583566 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:22.588077 (+ 4511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.589348 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.589373 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.589386 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.589674 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.590333 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.590352 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.590366 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.591031 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:22.591386 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.591982 (+ 596us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.592323 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[11739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[11739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503562, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 02:59:22.662532 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.638912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.10.118.65:39895)
0812 02:59:22.639327 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.639365 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.640901 (+ 1536us) server_negotiation.cc:407] Connection header received
0812 02:59:22.641321 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.641341 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.641513 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.641682 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:22.643872 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.644923 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.646903 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.647475 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.647749 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.652903 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:22.652931 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:22.652947 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:22.653079 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:22.657036 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.658107 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.658129 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.658142 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.658350 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.658997 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.659018 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.659031 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.659498 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:22.659923 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.660649 (+ 726us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.661098 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260812 02:59:22.693305 12680 tablet.cc:2369] T e2d5bb2097884ebbac158e661337568d P b5cc776aab1747a88c2597cc290e6569: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[11739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[11739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 02:59:22.731997 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:22.708369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57380 (local address 127.10.118.65:39895)
0812 02:59:22.709032 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:22.709078 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:22.710409 (+ 1331us) server_negotiation.cc:407] Connection header received
0812 02:59:22.710781 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:22.710799 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:22.710964 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:22.711098 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:22.713893 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.714896 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.716839 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:22.717555 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:22.717772 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:22.722977 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:22.723001 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:22.723017 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:22.723121 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:22.726785 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.727827 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.727855 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.727877 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.728188 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:22.728879 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:22.728900 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:22.728915 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:22.729477 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:22.729747 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:22.730382 (+ 635us) server_negotiation.cc:299] Negotiation successful
0812 02:59:22.730705 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260812 02:59:22.734980 12018 tablet_service.cc:2908] Rejecting scan request for tablet e2d5bb2097884ebbac158e661337568d: Uninitialized: clean time has not yet been initialized
I20260812 02:59:22.739245 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 11927
I20260812 02:59:22.774530 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 12188
I20260812 02:59:22.810832 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 12443
I20260812 02:59:22.844187 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 11767
2026-08-12T02:59:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (10156 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[12757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[12757](info): set up 2 sockets
Aug 12 02:59:22 dist-test-slave-zpvx krb5kdc[12757](info): commencing operation
krb5kdc: starting...
W20260812 02:59:24.386318 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.458s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260812 02:59:24.516999 10713 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.130s user 0.000s sys 0.008s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 02:59:24 dist-test-slave-zpvx krb5kdc[12757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:59:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:59:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 02:59:24.697049 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:40205
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:40205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:25.002791 12780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:25.003444 12780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:25.003997 12780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:25.034912 12780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:59:25.035198 12780 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:59:25.035378 12780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:25.035606 12780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:25.035799 12780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:59:25.035990 12780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:59:25.071607 12780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:40205
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:25.072743 12780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:25.073747 12780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:25.093040 12803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:25.093092 12797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:25.093215 12800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:25.094218 12780 server_base.cc:1034] running on GCE node
I20260812 02:59:26.301331 12780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:26.303885 12780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:26.305220 12780 hybrid_clock.cc:648] HybridClock initialized: now 1786503566305189 us; error 51 us; skew 500 ppm
I20260812 02:59:26.306069 12780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:26 dist-test-slave-zpvx krb5kdc[12757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503566, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:26.311883 12780 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 02:59:26.317044 12780 webserver.cc:460] Webserver started at http://127.10.118.126:42745/ using document root <none> and password file <none>
I20260812 02:59:26.317947 12780 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:26.318167 12780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:26.318655 12780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:26.323761 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "d5c74b38776942999ad850207e5da639"
format_stamp: "Formatted at 2026-08-12 02:59:26 on dist-test-slave-zpvx"
server_key: "137d6fc1def2a1db777c926c00309b28"
server_key_iv: "232b8401c03053d8fd4cbcb37c03fdff"
server_key_version: "encryptionkey@0"
I20260812 02:59:26.325027 12780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "d5c74b38776942999ad850207e5da639"
format_stamp: "Formatted at 2026-08-12 02:59:26 on dist-test-slave-zpvx"
server_key: "137d6fc1def2a1db777c926c00309b28"
server_key_iv: "232b8401c03053d8fd4cbcb37c03fdff"
server_key_version: "encryptionkey@0"
I20260812 02:59:26.332237 12780 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 02:59:26.337857 12817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:26.338866 12780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 02:59:26.339200 12780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "d5c74b38776942999ad850207e5da639"
format_stamp: "Formatted at 2026-08-12 02:59:26 on dist-test-slave-zpvx"
server_key: "137d6fc1def2a1db777c926c00309b28"
server_key_iv: "232b8401c03053d8fd4cbcb37c03fdff"
server_key_version: "encryptionkey@0"
I20260812 02:59:26.339576 12780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:26.370771 12780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:26.376207 12780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:26.376694 12780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:26.467231 12780 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:40205
I20260812 02:59:26.467375 12926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:40205 every 8 connection(s)
I20260812 02:59:26.470039 12780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 02:59:26.475184 12927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:26.478181 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 12780
I20260812 02:59:26.478632 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 02:59:26.479740 10713 external_mini_cluster.cc:1428] Setting key 395745ebf4d88bf15d56b8462a1ab102
I20260812 02:59:26.501641 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639: Bootstrap starting.
Aug 12 02:59:26 dist-test-slave-zpvx krb5kdc[12757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503564, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:26.507799 12927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:26.509802 12927 log.cc:824] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:26.515620 12927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639: No bootstrap required, opened a new log
I20260812 02:59:26.516619 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:26.482126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36358 (local address 127.10.118.126:40205)
0812 02:59:26.484711 (+ 2585us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:26.484751 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:26.484832 (+ 81us) server_negotiation.cc:407] Connection header received
0812 02:59:26.489144 (+ 4312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:26.489205 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:26.489711 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:26.490168 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:26.492396 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:26.495025 (+ 2629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:26.496993 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:26.498414 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:26.498747 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:26.504709 (+ 5962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:26.504733 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:26.504777 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:26.504878 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:26.509133 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:26.510584 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:26.510606 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:26.510622 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:26.510863 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:26.511746 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:26.511809 (+ 63us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:26.511835 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:26.512942 (+ 1107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:26.513442 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:26.514070 (+ 628us) server_negotiation.cc:299] Negotiation successful
0812 02:59:26.514497 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2006,"thread_start_us":1595,"threads_started":1}
I20260812 02:59:26.538425 12927 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c74b38776942999ad850207e5da639" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40205 } }
I20260812 02:59:26.539211 12927 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:26.539501 12927 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c74b38776942999ad850207e5da639, State: Initialized, Role: FOLLOWER
I20260812 02:59:26.540212 12927 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c74b38776942999ad850207e5da639" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40205 } }
I20260812 02:59:26.540820 12927 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:59:26.541118 12927 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:59:26.541435 12927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:26.545925 12927 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c74b38776942999ad850207e5da639" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40205 } }
I20260812 02:59:26.546850 12927 leader_election.cc:302] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c74b38776942999ad850207e5da639; no voters:
I20260812 02:59:26.548511 12927 leader_election.cc:288] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:59:26.548897 12939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:26.551502 12939 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [term 1 LEADER]: Becoming Leader. State: Replica: d5c74b38776942999ad850207e5da639, State: Running, Role: LEADER
I20260812 02:59:26.552241 12939 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c74b38776942999ad850207e5da639" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40205 } }
I20260812 02:59:26.553200 12927 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:59:26.563123 12941 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c74b38776942999ad850207e5da639. Latest consensus state: current_term: 1 leader_uuid: "d5c74b38776942999ad850207e5da639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c74b38776942999ad850207e5da639" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40205 } } }
I20260812 02:59:26.563781 12941 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:26.563031 12940 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c74b38776942999ad850207e5da639" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c74b38776942999ad850207e5da639" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40205 } } }
I20260812 02:59:26.564316 12940 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639 [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:26.566896 12951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:59:26.579041 12951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:59:26.596139 12951 catalog_manager.cc:1347] Generated new cluster ID: 2b865730d6794c8881188ac3c996122f
I20260812 02:59:26.596432 12951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:59:26.615595 12951 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:59:26.617345 12951 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:59:26.629956 12951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639: Generated new TSK 0
I20260812 02:59:26.630990 12951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 02:59:26.693514 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40205
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:27.001470 12971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:27.002028 12971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:27.002723 12971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:27.034967 12971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:27.035339 12971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:27.036103 12971 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:59:27.036684 12971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 02:59:27.071553 12971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:40205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:27.072785 12971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:27.073803 12971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:27.094431 12978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:27.096153 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:27.095275 12980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:27.096570 12971 server_base.cc:1034] running on GCE node
I20260812 02:59:28.282929 12971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:28.285935 12971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:28.287400 12971 hybrid_clock.cc:648] HybridClock initialized: now 1786503568287321 us; error 77 us; skew 500 ppm
I20260812 02:59:28.288633 12971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:28 dist-test-slave-zpvx krb5kdc[12757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:28.296459 12971 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 02:59:28.302556 12971 webserver.cc:460] Webserver started at http://127.10.118.65:39611/ using document root <none> and password file <none>
I20260812 02:59:28.303459 12971 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:28.303702 12971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:28.304162 12971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:28.309473 12971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "2936a02299544c36b6de6077366dc191"
format_stamp: "Formatted at 2026-08-12 02:59:28 on dist-test-slave-zpvx"
server_key: "0300d38baa38a5de934e303c0ba4d0e8"
server_key_iv: "b864233111dba9e46aac2385dd6a4457"
server_key_version: "encryptionkey@0"
I20260812 02:59:28.310649 12971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "2936a02299544c36b6de6077366dc191"
format_stamp: "Formatted at 2026-08-12 02:59:28 on dist-test-slave-zpvx"
server_key: "0300d38baa38a5de934e303c0ba4d0e8"
server_key_iv: "b864233111dba9e46aac2385dd6a4457"
server_key_version: "encryptionkey@0"
I20260812 02:59:28.318326 12971 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260812 02:59:28.323936 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:28.325012 12971 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 02:59:28.325354 12971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "2936a02299544c36b6de6077366dc191"
format_stamp: "Formatted at 2026-08-12 02:59:28 on dist-test-slave-zpvx"
server_key: "0300d38baa38a5de934e303c0ba4d0e8"
server_key_iv: "b864233111dba9e46aac2385dd6a4457"
server_key_version: "encryptionkey@0"
I20260812 02:59:28.325748 12971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:28.371778 12971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:28.378309 12971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:28.378775 12971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:28.388912 12971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:28.389225 12971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:28.389520 12971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:28.389695 12971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:59:28 dist-test-slave-zpvx krb5kdc[12757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503568, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:28.425402 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:28.385425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:36077 (local address 127.10.118.126:40205)
0812 02:59:28.386918 (+ 1493us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:28.386938 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:28.389086 (+ 2148us) server_negotiation.cc:407] Connection header received
0812 02:59:28.394987 (+ 5901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:28.395018 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:28.395279 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:28.395483 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:28.397927 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.399722 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.402566 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.403791 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.404166 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:28.410906 (+ 6740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:28.410949 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:28.410983 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:28.411172 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:28.417297 (+ 6125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:28.419262 (+ 1965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:28.419306 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:28.419340 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:28.419626 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:28.420506 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:28.420527 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:28.420545 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:28.421212 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:28.421477 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:28.422537 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0812 02:59:28.423028 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":979,"thread_start_us":632,"threads_started":1}
I20260812 02:59:28.426139 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:28.385532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40205 (local address 127.10.118.65:36077)
0812 02:59:28.388844 (+ 3312us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:28.388923 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:28.394613 (+ 5690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:28.395737 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:28.395761 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:28.396739 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:28.397636 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:28.397682 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.400027 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.400054 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:28.402177 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:28.402250 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.404201 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.404233 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:28.404938 (+ 705us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:28.406157 (+ 1219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:28.406301 (+ 144us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:28.410391 (+ 4090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:28.417569 (+ 7178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:28.417618 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:28.417669 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:28.418960 (+ 1291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:28.419895 (+ 935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:28.419932 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:28.419960 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:28.420245 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:28.421447 (+ 1202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:28.421499 (+ 52us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:28.421960 (+ 461us) client_negotiation.cc:769] Sending connection context
0812 02:59:28.422306 (+ 346us) client_negotiation.cc:240] Negotiation successful
0812 02:59:28.423061 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2776,"thread_start_us":1876,"threads_started":1}
I20260812 02:59:28.470841 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:28.442902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40205 (local address 127.10.118.65:39273)
0812 02:59:28.447083 (+ 4181us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:28.447157 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:28.447482 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:28.448434 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:28.448468 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:28.449882 (+ 1414us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:28.450865 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:28.450930 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.452829 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.452859 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:28.454579 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:28.454622 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.455764 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.455801 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:28.456205 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:28.457091 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:28.457155 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:28.458485 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:28.463577 (+ 5092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:28.463612 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:28.463637 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:28.464347 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:28.465158 (+ 811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:28.465190 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:28.465212 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:28.465494 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:28.466751 (+ 1257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:28.466778 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:28.467172 (+ 394us) client_negotiation.cc:769] Sending connection context
0812 02:59:28.467476 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 02:59:28.468051 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3607,"thread_start_us":2875,"threads_started":1}
I20260812 02:59:28.474511 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:28.442923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:39273 (local address 127.10.118.126:40205)
0812 02:59:28.444149 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:28.444180 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:28.447318 (+ 3138us) server_negotiation.cc:407] Connection header received
0812 02:59:28.447763 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:28.447804 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:28.448045 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:28.448297 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:28.451230 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.452599 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.454845 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:28.455426 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:28.455780 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:28.458811 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:28.458840 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:28.458881 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:28.459009 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:28.463360 (+ 4351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:28.464632 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:28.464661 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:28.464683 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:28.464923 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:28.465811 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:28.465838 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:28.465858 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:28.466513 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:28.471687 (+ 5174us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:28.471960 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 02:59:28.472481 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726}
I20260812 02:59:28.554201 12971 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:33981
I20260812 02:59:28.554903 13241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:33981 every 8 connection(s)
I20260812 02:59:28.556798 12971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 02:59:28.561444 13242 heartbeater.cc:347] Connected to a master server at 127.10.118.126:40205
I20260812 02:59:28.561914 13242 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:28.562942 13242 heartbeater.cc:511] Master 127.10.118.126:40205 requested a full tablet report, sending...
I20260812 02:59:28.563651 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 12971
I20260812 02:59:28.564078 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 02:59:28.565313 10713 external_mini_cluster.cc:1428] Setting key 292af9a180128ff4b9641a16218efac2
I20260812 02:59:28.565737 12851 ts_manager.cc:194] Registered new tserver with Master: 2936a02299544c36b6de6077366dc191 (127.10.118.65:33981)
I20260812 02:59:28.568926 12851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:36077
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 02:59:28.617089 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40205
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:28.925385 13250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:28.925866 13250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:28.926486 13250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:28.957789 13250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:28.958168 13250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:28.958839 13250 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:59:28.959324 13250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 02:59:28.994396 13250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:40205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:28.995496 13250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:28.996551 13250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:29.015925 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:29.572814 13242 heartbeater.cc:503] Master 127.10.118.126:40205 was elected leader, sending a full tablet report...
W20260812 02:59:29.015987 13265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:29.018339 13250 server_base.cc:1034] running on GCE node
W20260812 02:59:29.016798 13258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:30.207762 13250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:30.210639 13250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:30.211998 13250 hybrid_clock.cc:648] HybridClock initialized: now 1786503570211970 us; error 51 us; skew 500 ppm
I20260812 02:59:30.212929 13250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:30 dist-test-slave-zpvx krb5kdc[12757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503570, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:30.219282 13250 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 02:59:30.224489 13250 webserver.cc:460] Webserver started at http://127.10.118.66:41701/ using document root <none> and password file <none>
I20260812 02:59:30.225385 13250 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:30.225623 13250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:30.226086 13250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:30.231442 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "fb55a80c5d094ded836ac0eafd57c18d"
format_stamp: "Formatted at 2026-08-12 02:59:30 on dist-test-slave-zpvx"
server_key: "bd3f0ac392d26375c64af76d4b64d07f"
server_key_iv: "2cbdaee5a2c296dab40c24935f36a1a5"
server_key_version: "encryptionkey@0"
I20260812 02:59:30.232671 13250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "fb55a80c5d094ded836ac0eafd57c18d"
format_stamp: "Formatted at 2026-08-12 02:59:30 on dist-test-slave-zpvx"
server_key: "bd3f0ac392d26375c64af76d4b64d07f"
server_key_iv: "2cbdaee5a2c296dab40c24935f36a1a5"
server_key_version: "encryptionkey@0"
I20260812 02:59:30.239773 13250 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 02:59:30.245862 13275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:30.247011 13250 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 02:59:30.247349 13250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "fb55a80c5d094ded836ac0eafd57c18d"
format_stamp: "Formatted at 2026-08-12 02:59:30 on dist-test-slave-zpvx"
server_key: "bd3f0ac392d26375c64af76d4b64d07f"
server_key_iv: "2cbdaee5a2c296dab40c24935f36a1a5"
server_key_version: "encryptionkey@0"
I20260812 02:59:30.247737 13250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:30.306602 13250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:30.311965 13250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:30.312479 13250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:30.323566 13250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:30.323834 13250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:30.324110 13250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:30.324278 13250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:59:30 dist-test-slave-zpvx krb5kdc[12757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503570, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:30.355376 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:30.318981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:47161 (local address 127.10.118.126:40205)
0812 02:59:30.320484 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:30.320516 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:30.323059 (+ 2543us) server_negotiation.cc:407] Connection header received
0812 02:59:30.329101 (+ 6042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:30.329136 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:30.329316 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:30.329500 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:30.331776 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.332807 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.335777 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.336784 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.337089 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:30.344428 (+ 7339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:30.344467 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:30.344484 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:30.344615 (+ 131us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:30.348504 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:30.349687 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:30.349720 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:30.349734 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:30.349932 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:30.350626 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:30.350653 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:30.350667 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:30.351151 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:30.351390 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:30.352445 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0812 02:59:30.352920 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":625,"threads_started":1}
I20260812 02:59:30.356469 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:30.319276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40205 (local address 127.10.118.66:47161)
0812 02:59:30.322795 (+ 3519us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:30.322885 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:30.328727 (+ 5842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:30.329781 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:30.329817 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:30.330629 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:30.331493 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:30.331534 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.333062 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.333106 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:30.335526 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:30.335572 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.337151 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.337188 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:30.337880 (+ 692us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:30.339202 (+ 1322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:30.339335 (+ 133us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:30.344054 (+ 4719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:30.348708 (+ 4654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:30.348748 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:30.348774 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:30.349487 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:30.350134 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:30.350154 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:30.350170 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:30.350435 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:30.351382 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:30.351437 (+ 55us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:30.351859 (+ 422us) client_negotiation.cc:769] Sending connection context
0812 02:59:30.352200 (+ 341us) client_negotiation.cc:240] Negotiation successful
0812 02:59:30.352949 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3037,"thread_start_us":2176,"threads_started":1}
I20260812 02:59:30.413374 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:30.376744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:35007 (local address 127.10.118.126:40205)
0812 02:59:30.377527 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:30.377552 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:30.383093 (+ 5541us) server_negotiation.cc:407] Connection header received
0812 02:59:30.383636 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:30.383682 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:30.383985 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:30.384228 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:30.386969 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.388832 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.392261 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.393635 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.393995 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:30.398465 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:30.398508 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:30.398550 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:30.398754 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:30.405302 (+ 6548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:30.406971 (+ 1669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:30.407003 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:30.407031 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:30.407329 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:30.408337 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:30.408392 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:30.408418 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:30.409332 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:30.409668 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:30.410705 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0812 02:59:30.411181 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260812 02:59:30.413888 13370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:30.375524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40205 (local address 127.10.118.66:35007)
0812 02:59:30.382811 (+ 7287us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:30.382874 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:30.383300 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:30.384314 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:30.384356 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:30.385647 (+ 1291us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:30.386635 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:30.386684 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.389164 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.389202 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:30.391875 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:30.391953 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:30.394022 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:30.394063 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:30.394507 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:30.395701 (+ 1194us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:30.395792 (+ 91us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:30.397950 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:30.405573 (+ 7623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:30.405605 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:30.405631 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:30.406690 (+ 1059us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:30.407624 (+ 934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:30.407666 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:30.407699 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:30.408061 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:30.409630 (+ 1569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:30.409663 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:30.410098 (+ 435us) client_negotiation.cc:769] Sending connection context
0812 02:59:30.410444 (+ 346us) client_negotiation.cc:240] Negotiation successful
0812 02:59:30.410990 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6765,"thread_start_us":5087,"threads_started":1}
I20260812 02:59:30.494699 13250 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:33045
I20260812 02:59:30.495278 13523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:33045 every 8 connection(s)
I20260812 02:59:30.497258 13250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 02:59:30.499114 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 13250
I20260812 02:59:30.499699 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 02:59:30.501252 10713 external_mini_cluster.cc:1428] Setting key 971520e9b8f8495fec60dd47614efa55
I20260812 02:59:30.502647 13524 heartbeater.cc:347] Connected to a master server at 127.10.118.126:40205
I20260812 02:59:30.503116 13524 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:30.504266 13524 heartbeater.cc:511] Master 127.10.118.126:40205 requested a full tablet report, sending...
I20260812 02:59:30.506618 12851 ts_manager.cc:194] Registered new tserver with Master: fb55a80c5d094ded836ac0eafd57c18d (127.10.118.66:33045)
I20260812 02:59:30.508036 12851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:47161
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 02:59:30.553162 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40205
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:30.874548 13529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:30.875109 13529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:30.875788 13529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:30.906347 13529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:30.906804 13529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:30.907462 13529 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:59:30.907982 13529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 02:59:30.942951 13529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:40205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:30.944149 13529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:30.945217 13529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:30.964794 13543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:31.512730 13524 heartbeater.cc:503] Master 127.10.118.126:40205 was elected leader, sending a full tablet report...
W20260812 02:59:30.964787 13544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:30.966486 13529 server_base.cc:1034] running on GCE node
W20260812 02:59:30.965034 13549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:32.142625 13529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:32.145453 13529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:32.146957 13529 hybrid_clock.cc:648] HybridClock initialized: now 1786503572146900 us; error 74 us; skew 500 ppm
I20260812 02:59:32.147868 13529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:32 dist-test-slave-zpvx krb5kdc[12757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503572, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:32.153700 13529 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 02:59:32.158455 13529 webserver.cc:460] Webserver started at http://127.10.118.67:41135/ using document root <none> and password file <none>
I20260812 02:59:32.159364 13529 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:32.159624 13529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:32.160140 13529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:32.165894 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "1ac44449206647aa9cfb0a7f30b4592f"
format_stamp: "Formatted at 2026-08-12 02:59:32 on dist-test-slave-zpvx"
server_key: "bd0824ca3b6240b310d02b30818558b4"
server_key_iv: "60c97175e1022bc68e96be83f8ff9f10"
server_key_version: "encryptionkey@0"
I20260812 02:59:32.167064 13529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "1ac44449206647aa9cfb0a7f30b4592f"
format_stamp: "Formatted at 2026-08-12 02:59:32 on dist-test-slave-zpvx"
server_key: "bd0824ca3b6240b310d02b30818558b4"
server_key_iv: "60c97175e1022bc68e96be83f8ff9f10"
server_key_version: "encryptionkey@0"
I20260812 02:59:32.174430 13529 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260812 02:59:32.180269 13565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:32.181386 13529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 02:59:32.181712 13529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "1ac44449206647aa9cfb0a7f30b4592f"
format_stamp: "Formatted at 2026-08-12 02:59:32 on dist-test-slave-zpvx"
server_key: "bd0824ca3b6240b310d02b30818558b4"
server_key_iv: "60c97175e1022bc68e96be83f8ff9f10"
server_key_version: "encryptionkey@0"
I20260812 02:59:32.182103 13529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:32.249328 13529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:32.255537 13529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:32.256011 13529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:32.265872 13529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:32.266204 13529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:32.266578 13529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:32.266891 13529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:59:32 dist-test-slave-zpvx krb5kdc[12757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503572, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:32.305990 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.263088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:46355 (local address 127.10.118.126:40205)
0812 02:59:32.264639 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.264666 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.267992 (+ 3326us) server_negotiation.cc:407] Connection header received
0812 02:59:32.275912 (+ 7920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.275933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.276133 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.276300 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:32.279966 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.281289 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.285167 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.285877 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.286177 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.291352 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:32.291375 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:32.291391 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:32.291491 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:32.297457 (+ 5966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:32.298747 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:32.298768 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:32.298784 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:32.298983 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:32.299784 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:32.299827 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:32.299861 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:32.300704 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:32.301008 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.302729 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.303362 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":625,"threads_started":1}
I20260812 02:59:32.307102 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.263258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40205 (local address 127.10.118.67:46355)
0812 02:59:32.267648 (+ 4390us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:32.267767 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:32.275466 (+ 7699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:32.276671 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:32.276713 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:32.278014 (+ 1301us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:32.279536 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.279661 (+ 125us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.281606 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.281659 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.284819 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.284899 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.286237 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.286272 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.286745 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.287676 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:32.287747 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:32.291002 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:32.297735 (+ 6733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:32.297779 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:32.297819 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:32.298525 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:32.299197 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:32.299220 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:32.299235 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:32.299557 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:32.301004 (+ 1447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:32.301074 (+ 70us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:32.301827 (+ 753us) client_negotiation.cc:769] Sending connection context
0812 02:59:32.302393 (+ 566us) client_negotiation.cc:240] Negotiation successful
0812 02:59:32.303346 (+ 953us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3757,"thread_start_us":2861,"threads_started":1}
I20260812 02:59:32.366463 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.333247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:38467 (local address 127.10.118.126:40205)
0812 02:59:32.333912 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.333946 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.336512 (+ 2566us) server_negotiation.cc:407] Connection header received
0812 02:59:32.337010 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.337045 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.337304 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.337510 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:32.340120 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.341864 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.345417 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.346591 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.346969 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.351324 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:32.351362 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:32.351401 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:32.351566 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:32.357587 (+ 6021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:32.359335 (+ 1748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:32.359378 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:32.359410 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:32.359843 (+ 433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:32.360874 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:32.360911 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:32.360944 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:32.361764 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:32.362057 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.363237 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.363843 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260812 02:59:32.366571 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.333078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40205 (local address 127.10.118.67:38467)
0812 02:59:32.336221 (+ 3143us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:32.336307 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:32.336712 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:32.337700 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:32.337746 (+ 46us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:32.339119 (+ 1373us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:32.339781 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.339830 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.342208 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.342252 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.344892 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.344972 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.346968 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.347011 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.347531 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.348715 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:32.348807 (+ 92us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:32.350935 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:32.357864 (+ 6929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:32.357913 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:32.357953 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:32.358950 (+ 997us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:32.360104 (+ 1154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:32.360145 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:32.360171 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:32.360597 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:32.362065 (+ 1468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:32.362107 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:32.362608 (+ 501us) client_negotiation.cc:769] Sending connection context
0812 02:59:32.362941 (+ 333us) client_negotiation.cc:240] Negotiation successful
0812 02:59:32.363590 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2609,"thread_start_us":2099,"threads_started":1}
I20260812 02:59:32.439061 13529 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:40349
I20260812 02:59:32.439657 13781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:40349 every 8 connection(s)
I20260812 02:59:32.441963 13529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 02:59:32.447152 13782 heartbeater.cc:347] Connected to a master server at 127.10.118.126:40205
I20260812 02:59:32.447532 13782 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:32.448144 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 13529
I20260812 02:59:32.448570 13782 heartbeater.cc:511] Master 127.10.118.126:40205 requested a full tablet report, sending...
I20260812 02:59:32.448686 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 02:59:32.449805 10713 external_mini_cluster.cc:1428] Setting key 97220ee011486a993afa011aabaf729e
I20260812 02:59:32.450949 12851 ts_manager.cc:194] Registered new tserver with Master: 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349)
I20260812 02:59:32.452806 12851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:46355
I20260812 02:59:32.455389 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:59:32 dist-test-slave-zpvx krb5kdc[12757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 02:59:32 dist-test-slave-zpvx krb5kdc[12757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:32.541702 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.513424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.10.118.126:40205)
0812 02:59:32.513979 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.514017 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.515211 (+ 1194us) server_negotiation.cc:407] Connection header received
0812 02:59:32.515661 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.515682 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.515875 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.516049 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:32.518683 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.519608 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.521627 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.522243 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.522526 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.532699 (+ 10173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:32.532721 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:32.532740 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:32.532848 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:32.536130 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:32.537249 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:32.537273 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:32.537287 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:32.537489 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:32.538137 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:32.538160 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:32.538176 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:32.538688 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:32.539080 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.539755 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.540260 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 02:59:32.548045 12851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:59:32.575707 12851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:32.628649 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.602787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.10.118.65:33981)
0812 02:59:32.614574 (+ 11787us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.614605 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.614664 (+ 59us) server_negotiation.cc:407] Connection header received
0812 02:59:32.614913 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.614947 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.615334 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.616671 (+ 1337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:32.617172 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.618761 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.622894 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.624325 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.624689 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.625135 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.626359 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.626659 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11274,"thread_start_us":10920,"threads_started":1}
I20260812 02:59:32.628942 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.602385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33981 (local address 127.0.0.1:39686)
0812 02:59:32.610800 (+ 8415us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:32.610905 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:32.611361 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:32.615664 (+ 4303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:32.615696 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:32.615806 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:32.616322 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.616359 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.619052 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.619090 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.622615 (+ 3525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.622654 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.624699 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.624736 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.625942 (+ 1206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.626057 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 02:59:32.626257 (+ 200us) client_negotiation.cc:240] Negotiation successful
0812 02:59:32.626617 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5039,"mutex_wait_us":315,"thread_start_us":4692,"threads_started":1}
I20260812 02:59:32.631770 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.605464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.10.118.67:40349)
0812 02:59:32.615586 (+ 10122us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.615619 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.615676 (+ 57us) server_negotiation.cc:407] Connection header received
0812 02:59:32.615884 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.615921 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.616261 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.616533 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:32.617790 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.619656 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.624398 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.626445 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.626747 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.627219 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.629656 (+ 2437us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.629950 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9604,"thread_start_us":7738,"threads_started":1}
I20260812 02:59:32.631834 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.603796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:40349 (local address 127.0.0.1:45658)
0812 02:59:32.612409 (+ 8613us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:32.612721 (+ 312us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:32.613058 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:32.616569 (+ 3511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:32.616606 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:32.616702 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:32.617408 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.617459 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.619939 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.619971 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.624000 (+ 4029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.624053 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.628540 (+ 4487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.628576 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.629015 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.629233 (+ 218us) client_negotiation.cc:769] Sending connection context
0812 02:59:32.629574 (+ 341us) client_negotiation.cc:240] Negotiation successful
0812 02:59:32.629948 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7971,"mutex_wait_us":782,"thread_start_us":7622,"threads_started":1}
I20260812 02:59:32.634641 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.608507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:33045 (local address 127.0.0.1:44218)
0812 02:59:32.615237 (+ 6730us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:32.615299 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:32.615556 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:32.617100 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:32.617119 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:32.617199 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:32.617969 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.618034 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.620486 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.620521 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.624431 (+ 3910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.624480 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.628072 (+ 3592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.628106 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.632555 (+ 4449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.632712 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 02:59:32.632885 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 02:59:32.633195 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6274,"mutex_wait_us":729,"thread_start_us":5930,"threads_started":1}
I20260812 02:59:32.634796 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.608740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.10.118.66:33045)
0812 02:59:32.612737 (+ 3997us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.612792 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.615776 (+ 2984us) server_negotiation.cc:407] Connection header received
0812 02:59:32.615991 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.616022 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.616340 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.616556 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:32.618294 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.619712 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.624803 (+ 5091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.626659 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.626962 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.627349 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.632989 (+ 5640us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.633288 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3362,"thread_start_us":3152,"threads_started":1}
I20260812 02:59:32.633275 13111 tablet_service.cc:1467] Processing CreateTablet for tablet c727a3a0378b4a8ba11e47bc406a02ea (DEFAULT_TABLE table=test-table [id=d6765b17c20c4ffd82664fe5863f71cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:32.637081 13111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c727a3a0378b4a8ba11e47bc406a02ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:32.636906 13663 tablet_service.cc:1467] Processing CreateTablet for tablet c727a3a0378b4a8ba11e47bc406a02ea (DEFAULT_TABLE table=test-table [id=d6765b17c20c4ffd82664fe5863f71cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:32.638922 13663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c727a3a0378b4a8ba11e47bc406a02ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:32.639091 13400 tablet_service.cc:1467] Processing CreateTablet for tablet c727a3a0378b4a8ba11e47bc406a02ea (DEFAULT_TABLE table=test-table [id=d6765b17c20c4ffd82664fe5863f71cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:32.640954 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c727a3a0378b4a8ba11e47bc406a02ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:32.664233 13822 tablet_bootstrap.cc:492] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Bootstrap starting.
I20260812 02:59:32.670617 13824 tablet_bootstrap.cc:492] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Bootstrap starting.
I20260812 02:59:32.671443 13822 tablet_bootstrap.cc:654] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:32.674580 13822 log.cc:824] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:32.678258 13828 tablet_bootstrap.cc:492] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Bootstrap starting.
I20260812 02:59:32.679092 13824 tablet_bootstrap.cc:654] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:32.681838 13824 log.cc:824] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:32.682112 13822 tablet_bootstrap.cc:492] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: No bootstrap required, opened a new log
I20260812 02:59:32.682590 13822 ts_tablet_manager.cc:1397] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20260812 02:59:32.687553 13828 tablet_bootstrap.cc:654] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:32.688723 13824 tablet_bootstrap.cc:492] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: No bootstrap required, opened a new log
I20260812 02:59:32.689303 13824 ts_tablet_manager.cc:1397] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.005s
I20260812 02:59:32.690564 13828 log.cc:824] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:32.699632 13828 tablet_bootstrap.cc:492] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: No bootstrap required, opened a new log
I20260812 02:59:32.700343 13828 ts_tablet_manager.cc:1397] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20260812 02:59:32.714035 13824 raft_consensus.cc:348] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.715049 13824 raft_consensus.cc:374] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:32.715487 13824 raft_consensus.cc:729] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac44449206647aa9cfb0a7f30b4592f, State: Initialized, Role: FOLLOWER
I20260812 02:59:32.716451 13824 consensus_queue.cc:260] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.721191 13822 raft_consensus.cc:348] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.722204 13822 raft_consensus.cc:374] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:32.722585 13822 raft_consensus.cc:729] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2936a02299544c36b6de6077366dc191, State: Initialized, Role: FOLLOWER
I20260812 02:59:32.723621 13822 consensus_queue.cc:260] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.724956 13782 heartbeater.cc:503] Master 127.10.118.126:40205 was elected leader, sending a full tablet report...
I20260812 02:59:32.725924 13824 ts_tablet_manager.cc:1428] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Time spent starting tablet: real 0.036s user 0.026s sys 0.009s
I20260812 02:59:32.735173 13822 ts_tablet_manager.cc:1428] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Time spent starting tablet: real 0.052s user 0.042s sys 0.008s
I20260812 02:59:32.737126 13828 raft_consensus.cc:348] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.738058 13828 raft_consensus.cc:374] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:32.738390 13828 raft_consensus.cc:729] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb55a80c5d094ded836ac0eafd57c18d, State: Initialized, Role: FOLLOWER
I20260812 02:59:32.739326 13828 consensus_queue.cc:260] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.745193 13828 ts_tablet_manager.cc:1428] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Time spent starting tablet: real 0.045s user 0.033s sys 0.009s
W20260812 02:59:32.754171 13525 tablet.cc:2369] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:59:32.810982 13844 raft_consensus.cc:482] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:32.811728 13844 raft_consensus.cc:504] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
W20260812 02:59:32.815446 13244 tablet.cc:2369] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:59:32.814661 13844 leader_election.cc:288] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2936a02299544c36b6de6077366dc191 (127.10.118.65:33981), 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349)
I20260812 02:59:32.831532 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.815169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:42011 (local address 127.10.118.65:33981)
0812 02:59:32.815919 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.815948 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.820156 (+ 4208us) server_negotiation.cc:407] Connection header received
0812 02:59:32.820641 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.820669 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.820946 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.821070 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:32.822491 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.823554 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.826828 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.828340 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.828743 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.829007 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.829630 (+ 623us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.829949 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260812 02:59:32.831676 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.815016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33981 (local address 127.10.118.66:42011)
0812 02:59:32.819965 (+ 4949us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:32.820011 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:32.820324 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:32.821161 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:32.821191 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:32.821296 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:32.822183 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.822232 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.823828 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.823868 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.826563 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.826602 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.828728 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.828768 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.829206 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.829350 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 02:59:32.829508 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 02:59:32.829790 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4436,"mutex_wait_us":331,"thread_start_us":4311,"threads_started":1}
I20260812 02:59:32.832701 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.816408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:54795 (local address 127.10.118.67:40349)
0812 02:59:32.817095 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.817136 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.820549 (+ 3413us) server_negotiation.cc:407] Connection header received
0812 02:59:32.820911 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.820929 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.821171 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.821353 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:32.823045 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.824081 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.827832 (+ 3751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.829250 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.829635 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.829894 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.830843 (+ 949us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.831212 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260812 02:59:32.833061 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.815162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:40349 (local address 127.10.118.66:54795)
0812 02:59:32.820317 (+ 5155us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:32.820361 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:32.820688 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:32.821434 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:32.821457 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:32.821545 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:32.822710 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.822759 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.824490 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.824512 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.827561 (+ 3049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:32.827611 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.829851 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.829888 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:32.830402 (+ 514us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.830591 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 02:59:32.830763 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 02:59:32.831106 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4580,"mutex_wait_us":235,"thread_start_us":3538,"threads_started":1}
I20260812 02:59:32.834616 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c727a3a0378b4a8ba11e47bc406a02ea" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac44449206647aa9cfb0a7f30b4592f" is_pre_election: true
I20260812 02:59:32.834620 13148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c727a3a0378b4a8ba11e47bc406a02ea" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2936a02299544c36b6de6077366dc191" is_pre_election: true
I20260812 02:59:32.835582 13694 raft_consensus.cc:2454] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 0.
I20260812 02:59:32.835692 13148 raft_consensus.cc:2454] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 0.
I20260812 02:59:32.837090 13279 leader_election.cc:302] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac44449206647aa9cfb0a7f30b4592f, fb55a80c5d094ded836ac0eafd57c18d; no voters:
I20260812 02:59:32.838019 13844 raft_consensus.cc:2781] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:32.838426 13844 raft_consensus.cc:482] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:32.838696 13844 raft_consensus.cc:3037] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:32.844331 13844 raft_consensus.cc:504] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.846657 13844 leader_election.cc:288] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 election: Requested vote from peers 2936a02299544c36b6de6077366dc191 (127.10.118.65:33981), 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349)
I20260812 02:59:32.847425 13148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c727a3a0378b4a8ba11e47bc406a02ea" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2936a02299544c36b6de6077366dc191"
I20260812 02:59:32.847709 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c727a3a0378b4a8ba11e47bc406a02ea" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac44449206647aa9cfb0a7f30b4592f"
I20260812 02:59:32.847906 13148 raft_consensus.cc:3037] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:32.848219 13694 raft_consensus.cc:3037] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:32.853287 13148 raft_consensus.cc:2454] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 1.
I20260812 02:59:32.853291 13694 raft_consensus.cc:2454] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 1.
I20260812 02:59:32.854255 13279 leader_election.cc:302] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac44449206647aa9cfb0a7f30b4592f, fb55a80c5d094ded836ac0eafd57c18d; no voters:
I20260812 02:59:32.854990 13844 raft_consensus.cc:2781] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:32.856900 13844 raft_consensus.cc:686] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 1 LEADER]: Becoming Leader. State: Replica: fb55a80c5d094ded836ac0eafd57c18d, State: Running, Role: LEADER
I20260812 02:59:32.857838 13844 consensus_queue.cc:237] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } }
I20260812 02:59:32.872051 12849 catalog_manager.cc:5495] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb55a80c5d094ded836ac0eafd57c18d (127.10.118.66). New cstate: current_term: 1 leader_uuid: "fb55a80c5d094ded836ac0eafd57c18d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } health_report { overall_health: HEALTHY } } }
W20260812 02:59:32.946720 13783 tablet.cc:2369] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:59:32.948112 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:32.937510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.10.118.66:33045)
0812 02:59:32.938133 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:32.938154 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:32.938363 (+ 209us) server_negotiation.cc:407] Connection header received
0812 02:59:32.938920 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:32.938940 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:32.939180 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:32.939329 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:32.940286 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.941318 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.943516 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:32.944144 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:32.944358 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:32.945442 (+ 1084us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:32.945961 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:32.946100 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:32.946418 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 02:59:32.946739 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 02:59:32.963565 13694 raft_consensus.cc:1261] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Refusing update from remote peer fb55a80c5d094ded836ac0eafd57c18d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:32.964075 13148 raft_consensus.cc:1261] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Refusing update from remote peer fb55a80c5d094ded836ac0eafd57c18d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:32.965309 13849 consensus_queue.cc:1035] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:32.965941 13844 consensus_queue.cc:1035] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:33.001490 13860 mvcc.cc:205] Tried to move back new op lower bound from 7317518634840477696 to 7317518634435686400. Current Snapshot: MvccSnapshot[applied={T|T < 7317518634840477696}]
I20260812 02:59:33.038758 12851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:33.039438 12851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:33.050804 12851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639: Sending DeleteTablet for 3 replicas of tablet c727a3a0378b4a8ba11e47bc406a02ea
I20260812 02:59:33.052970 13111 tablet_service.cc:1514] Processing DeleteTablet for tablet c727a3a0378b4a8ba11e47bc406a02ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:33 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:39686
I20260812 02:59:33.053092 13663 tablet_service.cc:1514] Processing DeleteTablet for tablet c727a3a0378b4a8ba11e47bc406a02ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:33 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:45658
I20260812 02:59:33.055409 13400 tablet_service.cc:1514] Processing DeleteTablet for tablet c727a3a0378b4a8ba11e47bc406a02ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:33 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:44218
I20260812 02:59:33.063020 12851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:59:33.065822 13876 tablet_replica.cc:330] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: stopping tablet replica
I20260812 02:59:33.067502 13876 raft_consensus.cc:2229] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:33.068308 13876 raft_consensus.cc:2258] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:33.069128 13874 tablet_replica.cc:330] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: stopping tablet replica
W20260812 02:59:33.069200 12851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:33.071167 13874 raft_consensus.cc:2229] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:33.071964 13877 tablet_replica.cc:330] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: stopping tablet replica
I20260812 02:59:33.071929 13874 raft_consensus.cc:2258] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:33.072817 13876 ts_tablet_manager.cc:1905] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:33.073540 13877 raft_consensus.cc:2229] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:59:33.074699 13877 raft_consensus.cc:2258] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:33.076802 13874 ts_tablet_manager.cc:1905] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:33.078339 13877 ts_tablet_manager.cc:1905] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:33.094911 13877 ts_tablet_manager.cc:1918] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:59:33.095548 13877 log.cc:1196] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/c727a3a0378b4a8ba11e47bc406a02ea
I20260812 02:59:33.096587 13877 ts_tablet_manager.cc:1939] T c727a3a0378b4a8ba11e47bc406a02ea P fb55a80c5d094ded836ac0eafd57c18d: Deleting consensus metadata
I20260812 02:59:33.097468 13111 tablet_service.cc:1467] Processing CreateTablet for tablet 118d358deee04ac0bb34a7f98d7da325 (DEFAULT_TABLE table=test-table [id=55f8640a60124ff3af41acb1d53c8ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:33.098587 13111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118d358deee04ac0bb34a7f98d7da325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:33.100231 12818 catalog_manager.cc:4841] TS fb55a80c5d094ded836ac0eafd57c18d (127.10.118.66:33045): tablet c727a3a0378b4a8ba11e47bc406a02ea (table test-table [id=d6765b17c20c4ffd82664fe5863f71cc]) successfully deleted
I20260812 02:59:33.105321 13400 tablet_service.cc:1467] Processing CreateTablet for tablet 118d358deee04ac0bb34a7f98d7da325 (DEFAULT_TABLE table=test-table [id=55f8640a60124ff3af41acb1d53c8ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:33.106033 13663 tablet_service.cc:1467] Processing CreateTablet for tablet 118d358deee04ac0bb34a7f98d7da325 (DEFAULT_TABLE table=test-table [id=55f8640a60124ff3af41acb1d53c8ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:33.106493 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118d358deee04ac0bb34a7f98d7da325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:33.106774 13876 ts_tablet_manager.cc:1918] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:59:33.107189 13876 log.cc:1196] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/c727a3a0378b4a8ba11e47bc406a02ea
I20260812 02:59:33.107245 13663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118d358deee04ac0bb34a7f98d7da325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:33.108014 13876 ts_tablet_manager.cc:1939] T c727a3a0378b4a8ba11e47bc406a02ea P 2936a02299544c36b6de6077366dc191: Deleting consensus metadata
I20260812 02:59:33.113999 13874 ts_tablet_manager.cc:1918] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 02:59:33.115445 12818 catalog_manager.cc:4841] TS 2936a02299544c36b6de6077366dc191 (127.10.118.65:33981): tablet c727a3a0378b4a8ba11e47bc406a02ea (table test-table [id=d6765b17c20c4ffd82664fe5863f71cc]) successfully deleted
I20260812 02:59:33.116216 13874 log.cc:1196] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/c727a3a0378b4a8ba11e47bc406a02ea
I20260812 02:59:33.117347 13874 ts_tablet_manager.cc:1939] T c727a3a0378b4a8ba11e47bc406a02ea P 1ac44449206647aa9cfb0a7f30b4592f: Deleting consensus metadata
I20260812 02:59:33.122159 12824 catalog_manager.cc:4841] TS 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349): tablet c727a3a0378b4a8ba11e47bc406a02ea (table test-table [id=d6765b17c20c4ffd82664fe5863f71cc]) successfully deleted
I20260812 02:59:33.126983 13828 tablet_bootstrap.cc:492] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: Bootstrap starting.
I20260812 02:59:33.128676 13822 tablet_bootstrap.cc:492] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: Bootstrap starting.
I20260812 02:59:33.132038 13824 tablet_bootstrap.cc:492] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: Bootstrap starting.
I20260812 02:59:33.136823 13822 tablet_bootstrap.cc:654] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:33.139024 13824 tablet_bootstrap.cc:654] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:33.141227 13828 tablet_bootstrap.cc:654] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:33.150054 13822 tablet_bootstrap.cc:492] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: No bootstrap required, opened a new log
I20260812 02:59:33.150533 13822 ts_tablet_manager.cc:1397] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.009s
I20260812 02:59:33.153554 13822 raft_consensus.cc:348] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.154536 13824 tablet_bootstrap.cc:492] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: No bootstrap required, opened a new log
I20260812 02:59:33.154421 13822 raft_consensus.cc:374] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:33.154775 13822 raft_consensus.cc:729] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2936a02299544c36b6de6077366dc191, State: Initialized, Role: FOLLOWER
I20260812 02:59:33.154963 13824 ts_tablet_manager.cc:1397] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.003s
I20260812 02:59:33.155313 13828 tablet_bootstrap.cc:492] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: No bootstrap required, opened a new log
I20260812 02:59:33.155704 13828 ts_tablet_manager.cc:1397] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.006s
I20260812 02:59:33.155503 13822 consensus_queue.cc:260] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.158345 13824 raft_consensus.cc:348] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.158764 13828 raft_consensus.cc:348] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.159241 13824 raft_consensus.cc:374] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:33.159574 13828 raft_consensus.cc:374] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:33.159680 13824 raft_consensus.cc:729] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac44449206647aa9cfb0a7f30b4592f, State: Initialized, Role: FOLLOWER
I20260812 02:59:33.160084 13828 raft_consensus.cc:729] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb55a80c5d094ded836ac0eafd57c18d, State: Initialized, Role: FOLLOWER
I20260812 02:59:33.160804 13828 consensus_queue.cc:260] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.160848 13824 consensus_queue.cc:260] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.163022 13828 ts_tablet_manager.cc:1428] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260812 02:59:33.163935 13822 ts_tablet_manager.cc:1428] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20260812 02:59:33.165390 13824 ts_tablet_manager.cc:1428] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260812 02:59:33.318712 13849 raft_consensus.cc:482] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:33.319227 13849 raft_consensus.cc:504] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.321246 13849 leader_election.cc:288] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2936a02299544c36b6de6077366dc191 (127.10.118.65:33981), 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349)
I20260812 02:59:33.322239 13148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118d358deee04ac0bb34a7f98d7da325" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2936a02299544c36b6de6077366dc191" is_pre_election: true
I20260812 02:59:33.322474 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118d358deee04ac0bb34a7f98d7da325" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac44449206647aa9cfb0a7f30b4592f" is_pre_election: true
I20260812 02:59:33.322746 13148 raft_consensus.cc:2454] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 0.
I20260812 02:59:33.323130 13694 raft_consensus.cc:2454] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 0.
I20260812 02:59:33.323807 13276 leader_election.cc:302] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2936a02299544c36b6de6077366dc191, fb55a80c5d094ded836ac0eafd57c18d; no voters:
I20260812 02:59:33.324631 13849 raft_consensus.cc:2781] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:33.324945 13849 raft_consensus.cc:482] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:33.325263 13849 raft_consensus.cc:3037] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:33.330781 13849 raft_consensus.cc:504] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.332299 13849 leader_election.cc:288] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 election: Requested vote from peers 2936a02299544c36b6de6077366dc191 (127.10.118.65:33981), 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349)
I20260812 02:59:33.333197 13148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118d358deee04ac0bb34a7f98d7da325" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2936a02299544c36b6de6077366dc191"
I20260812 02:59:33.333525 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118d358deee04ac0bb34a7f98d7da325" candidate_uuid: "fb55a80c5d094ded836ac0eafd57c18d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac44449206647aa9cfb0a7f30b4592f"
I20260812 02:59:33.333817 13148 raft_consensus.cc:3037] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:33.334008 13694 raft_consensus.cc:3037] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:33.338703 13694 raft_consensus.cc:2454] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 1.
I20260812 02:59:33.339578 13279 leader_election.cc:302] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ac44449206647aa9cfb0a7f30b4592f, fb55a80c5d094ded836ac0eafd57c18d; no voters:
I20260812 02:59:33.340210 13849 raft_consensus.cc:2781] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:33.340665 13849 raft_consensus.cc:686] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 1 LEADER]: Becoming Leader. State: Replica: fb55a80c5d094ded836ac0eafd57c18d, State: Running, Role: LEADER
I20260812 02:59:33.341245 13148 raft_consensus.cc:2454] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb55a80c5d094ded836ac0eafd57c18d in term 1.
I20260812 02:59:33.341485 13849 consensus_queue.cc:237] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.351198 12849 catalog_manager.cc:5495] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb55a80c5d094ded836ac0eafd57c18d (127.10.118.66). New cstate: current_term: 1 leader_uuid: "fb55a80c5d094ded836ac0eafd57c18d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } health_report { overall_health: UNKNOWN } } }
Aug 12 02:59:33 dist-test-slave-zpvx krb5kdc[12757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503566, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:33.484308 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.437715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40205 (local address 127.0.0.1:36388)
0812 02:59:33.448357 (+ 10642us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:33.448437 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:33.448736 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:33.450909 (+ 2173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:33.450942 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:33.452313 (+ 1371us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:33.453268 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.453333 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.455728 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.455764 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.457967 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.458025 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.459970 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.460146 (+ 176us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.460760 (+ 614us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.462000 (+ 1240us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:33.462098 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:33.467121 (+ 5023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:33.475440 (+ 8319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:33.475492 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:33.475524 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:33.476715 (+ 1191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:33.477646 (+ 931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:33.477685 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:33.477713 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:33.478074 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:33.479616 (+ 1542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:33.479655 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:33.480151 (+ 496us) client_negotiation.cc:769] Sending connection context
0812 02:59:33.480520 (+ 369us) client_negotiation.cc:240] Negotiation successful
0812 02:59:33.481354 (+ 834us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8703,"mutex_wait_us":96,"thread_start_us":7882,"threads_started":1}
I20260812 02:59:33.490272 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.437190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.10.118.126:40205)
0812 02:59:33.449984 (+ 12794us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:33.450019 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:33.450075 (+ 56us) server_negotiation.cc:407] Connection header received
0812 02:59:33.450231 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:33.450260 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:33.450499 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:33.452504 (+ 2005us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:33.453618 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.455422 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.458280 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.459456 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.459911 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.467568 (+ 7657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:33.467604 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:33.467640 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:33.467802 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:33.475182 (+ 7380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:33.477040 (+ 1858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:33.477075 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:33.477102 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:33.477404 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:33.478411 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:33.478456 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:33.478489 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:33.479340 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:33.485086 (+ 5746us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:33.485610 (+ 524us) server_negotiation.cc:299] Negotiation successful
0812 02:59:33.486204 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":12210,"thread_start_us":11939,"threads_started":1}
I20260812 02:59:33.502753 12849 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:36388:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:59:33.506923 12849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:33.544351 13111 tablet_service.cc:1467] Processing CreateTablet for tablet b32feeeb159a42ae8a150a80829800d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc318c14680b421b9d226aeeb3b2a890]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:59:33.545998 13111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32feeeb159a42ae8a150a80829800d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:33.556586 13400 tablet_service.cc:1467] Processing CreateTablet for tablet b32feeeb159a42ae8a150a80829800d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc318c14680b421b9d226aeeb3b2a890]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:59:33.558166 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32feeeb159a42ae8a150a80829800d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:33.567556 13663 tablet_service.cc:1467] Processing CreateTablet for tablet b32feeeb159a42ae8a150a80829800d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc318c14680b421b9d226aeeb3b2a890]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:59:33.569089 13663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32feeeb159a42ae8a150a80829800d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:33.610847 13828 tablet_bootstrap.cc:492] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: Bootstrap starting.
I20260812 02:59:33.620286 13828 tablet_bootstrap.cc:654] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:33.625301 13822 tablet_bootstrap.cc:492] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: Bootstrap starting.
I20260812 02:59:33.634298 13822 tablet_bootstrap.cc:654] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:33.650138 13824 tablet_bootstrap.cc:492] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: Bootstrap starting.
I20260812 02:59:33.654456 13828 tablet_bootstrap.cc:492] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: No bootstrap required, opened a new log
I20260812 02:59:33.654927 13828 ts_tablet_manager.cc:1397] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.000s
I20260812 02:59:33.657672 13822 tablet_bootstrap.cc:492] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: No bootstrap required, opened a new log
I20260812 02:59:33.658119 13822 ts_tablet_manager.cc:1397] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.008s
I20260812 02:59:33.660959 13824 tablet_bootstrap.cc:654] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:33.660571 13822 raft_consensus.cc:348] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.661304 13822 raft_consensus.cc:374] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:33.661614 13822 raft_consensus.cc:729] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2936a02299544c36b6de6077366dc191, State: Initialized, Role: FOLLOWER
I20260812 02:59:33.662242 13822 consensus_queue.cc:260] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.668512 13828 raft_consensus.cc:348] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.669800 13828 raft_consensus.cc:374] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:33.670300 13828 raft_consensus.cc:729] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb55a80c5d094ded836ac0eafd57c18d, State: Initialized, Role: FOLLOWER
I20260812 02:59:33.671290 13828 consensus_queue.cc:260] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.672247 13824 tablet_bootstrap.cc:492] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: No bootstrap required, opened a new log
I20260812 02:59:33.672751 13824 ts_tablet_manager.cc:1397] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.008s
I20260812 02:59:33.672997 13849 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:33.673986 13849 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:33.678012 13822 ts_tablet_manager.cc:1428] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: Time spent starting tablet: real 0.020s user 0.011s sys 0.007s
I20260812 02:59:33.678058 13828 ts_tablet_manager.cc:1428] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260812 02:59:33.675621 13824 raft_consensus.cc:348] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.686664 13824 raft_consensus.cc:374] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:33.687011 13824 raft_consensus.cc:729] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac44449206647aa9cfb0a7f30b4592f, State: Initialized, Role: FOLLOWER
I20260812 02:59:33.687274 13908 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:33.688097 13908 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:33.688035 13824 consensus_queue.cc:260] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.690279 13912 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:33.691123 13912 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:33.692505 13824 ts_tablet_manager.cc:1428] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260812 02:59:33.816946 13908 raft_consensus.cc:482] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:33.817466 13908 raft_consensus.cc:504] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.820561 13908 leader_election.cc:288] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb55a80c5d094ded836ac0eafd57c18d (127.10.118.66:33045), 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349)
I20260812 02:59:33.847419 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.820948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:33045 (local address 127.10.118.65:47001)
0812 02:59:33.834349 (+ 13401us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:33.834416 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:33.834863 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:33.836023 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:33.836051 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:33.836154 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:33.836982 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.837035 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.838829 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.838851 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.841674 (+ 2823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.841729 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.844224 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.844258 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.844673 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.844807 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 02:59:33.844966 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 02:59:33.845340 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12844,"thread_start_us":12518,"threads_started":1}
I20260812 02:59:33.851125 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.821745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:47001 (local address 127.10.118.66:33045)
0812 02:59:33.835123 (+ 13378us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:33.835146 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:33.835200 (+ 54us) server_negotiation.cc:407] Connection header received
0812 02:59:33.835370 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:33.835402 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:33.835743 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:33.835940 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:33.837371 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.838565 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.842095 (+ 3530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.843891 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.844268 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.844625 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:33.845238 (+ 613us) server_negotiation.cc:299] Negotiation successful
0812 02:59:33.849227 (+ 3989us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12663,"thread_start_us":12350,"threads_started":1}
I20260812 02:59:33.851064 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.821768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:47153 (local address 127.10.118.67:40349)
0812 02:59:33.833767 (+ 11999us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:33.833802 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:33.836688 (+ 2886us) server_negotiation.cc:407] Connection header received
0812 02:59:33.837082 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:33.837102 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:33.837392 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:33.837588 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:33.839445 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.841037 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.844602 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.846316 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.846679 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.847057 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:33.848972 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0812 02:59:33.849347 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11503,"thread_start_us":11136,"threads_started":1}
I20260812 02:59:33.852661 13436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32feeeb159a42ae8a150a80829800d5" candidate_uuid: "2936a02299544c36b6de6077366dc191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb55a80c5d094ded836ac0eafd57c18d" is_pre_election: true
I20260812 02:59:33.853328 13436 raft_consensus.cc:2454] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2936a02299544c36b6de6077366dc191 in term 0.
I20260812 02:59:33.853609 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.821111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:40349 (local address 127.10.118.65:47153)
0812 02:59:33.836304 (+ 15193us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:33.836428 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:33.836763 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:33.837645 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:33.837675 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:33.837781 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:33.839103 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.839158 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.841389 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.841421 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.844225 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.844275 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.847326 (+ 3051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.847358 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.847789 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.847934 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 02:59:33.848096 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 02:59:33.848436 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14614,"mutex_wait_us":65,"thread_start_us":14226,"threads_started":1}
I20260812 02:59:33.854913 13005 leader_election.cc:302] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2936a02299544c36b6de6077366dc191, fb55a80c5d094ded836ac0eafd57c18d; no voters:
I20260812 02:59:33.855393 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32feeeb159a42ae8a150a80829800d5" candidate_uuid: "2936a02299544c36b6de6077366dc191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac44449206647aa9cfb0a7f30b4592f" is_pre_election: true
I20260812 02:59:33.855923 13908 raft_consensus.cc:2781] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:33.856158 13694 raft_consensus.cc:2454] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2936a02299544c36b6de6077366dc191 in term 0.
I20260812 02:59:33.856279 13908 raft_consensus.cc:482] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:33.856627 13908 raft_consensus.cc:3037] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:33.859229 13849 consensus_queue.cc:1035] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 02:59:33.864066 13908 raft_consensus.cc:504] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.867048 13436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32feeeb159a42ae8a150a80829800d5" candidate_uuid: "2936a02299544c36b6de6077366dc191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb55a80c5d094ded836ac0eafd57c18d"
I20260812 02:59:33.867772 13436 raft_consensus.cc:3037] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:33.875499 13436 raft_consensus.cc:2454] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2936a02299544c36b6de6077366dc191 in term 1.
I20260812 02:59:33.876797 13005 leader_election.cc:302] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2936a02299544c36b6de6077366dc191, fb55a80c5d094ded836ac0eafd57c18d; no voters:
I20260812 02:59:33.877424 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32feeeb159a42ae8a150a80829800d5" candidate_uuid: "2936a02299544c36b6de6077366dc191" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ac44449206647aa9cfb0a7f30b4592f"
I20260812 02:59:33.878104 13694 raft_consensus.cc:3037] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:33.866319 13908 leader_election.cc:288] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [CANDIDATE]: Term 1 election: Requested vote from peers fb55a80c5d094ded836ac0eafd57c18d (127.10.118.66:33045), 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349)
I20260812 02:59:33.884866 13908 raft_consensus.cc:2781] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:33.885563 13908 raft_consensus.cc:686] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [term 1 LEADER]: Becoming Leader. State: Replica: 2936a02299544c36b6de6077366dc191, State: Running, Role: LEADER
I20260812 02:59:33.886799 13908 consensus_queue.cc:237] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } }
I20260812 02:59:33.894515 13694 raft_consensus.cc:2454] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2936a02299544c36b6de6077366dc191 in term 1.
I20260812 02:59:33.898718 13924 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: TxnStatusTablet state changed. Reason: New leader 2936a02299544c36b6de6077366dc191. Latest consensus state: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:33.899645 13924 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:59:33.900652 12851 catalog_manager.cc:5495] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2936a02299544c36b6de6077366dc191 (127.10.118.65). New cstate: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } health_report { overall_health: UNKNOWN } } }
I20260812 02:59:33.914574 13926 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:59:33.915184 13926 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:59:33.947271 13923 consensus_queue.cc:1035] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 02:59:33.989748 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.958146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33981 (local address 127.0.0.1:39696)
0812 02:59:33.958782 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:33.958837 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:33.959181 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:33.974668 (+ 15487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:33.974705 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:33.974758 (+ 53us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:59:33.975207 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.975258 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.980812 (+ 5554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.980840 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.982953 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:33.982993 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.985128 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.985160 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:33.985566 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.985765 (+ 199us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:59:33.987592 (+ 1827us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:59:33.987643 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 02:59:33.987765 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 02:59:33.988093 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 02:59:33.992324 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:33.959065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.10.118.65:33981)
0812 02:59:33.973628 (+ 14563us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:33.973659 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:33.973711 (+ 52us) server_negotiation.cc:407] Connection header received
0812 02:59:33.973901 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:33.973929 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:33.974331 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:33.974620 (+ 289us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:33.976255 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.980538 (+ 4283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.983641 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:33.984745 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:33.986292 (+ 1547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:33.986662 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:33.987368 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:33.990465 (+ 3097us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:33.990557 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 02:59:33.990693 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14041,"spinlock_wait_cycles":37632,"thread_start_us":13985,"threads_started":1}
I20260812 02:59:33.999387 13694 raft_consensus.cc:1261] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Refusing update from remote peer 2936a02299544c36b6de6077366dc191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:34.000145 13436 raft_consensus.cc:1261] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Refusing update from remote peer 2936a02299544c36b6de6077366dc191: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:34.001804 13908 consensus_queue.cc:1035] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:34.003388 13908 consensus_queue.cc:1035] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:34.022722 13912 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: TxnStatusTablet state changed. Reason: New leader 2936a02299544c36b6de6077366dc191. Latest consensus state: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:34.023595 13912 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:34.033648 13923 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: TxnStatusTablet state changed. Reason: New leader 2936a02299544c36b6de6077366dc191. Latest consensus state: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:34.034399 13923 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:34.039418 13934 mvcc.cc:205] Tried to move back new op lower bound from 7317518639088058368 to 7317518638653259776. Current Snapshot: MvccSnapshot[applied={T|T < 7317518639088058368}]
I20260812 02:59:34.041071 13927 mvcc.cc:205] Tried to move back new op lower bound from 7317518639088058368 to 7317518638653259776. Current Snapshot: MvccSnapshot[applied={T|T < 7317518639088058368}]
I20260812 02:59:34.047672 13908 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:34.050709 13908 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:59:34.052050 13908 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:34.054437 13908 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P 2936a02299544c36b6de6077366dc191: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:59:34.064479 13912 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:34.065191 13912 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P 1ac44449206647aa9cfb0a7f30b4592f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:34.078094 13923 tablet_replica.cc:439] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2936a02299544c36b6de6077366dc191" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2936a02299544c36b6de6077366dc191" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33981 } } peers { permanent_uuid: "fb55a80c5d094ded836ac0eafd57c18d" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33045 } } peers { permanent_uuid: "1ac44449206647aa9cfb0a7f30b4592f" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 40349 } } }
I20260812 02:59:34.078850 13923 tablet_replica.cc:442] T b32feeeb159a42ae8a150a80829800d5 P fb55a80c5d094ded836ac0eafd57c18d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:34.110322 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:34.095210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:58443 (local address 127.10.118.65:33981)
0812 02:59:34.095777 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:34.095804 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:34.098146 (+ 2342us) server_negotiation.cc:407] Connection header received
0812 02:59:34.098658 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:34.098690 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:34.099056 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:34.099242 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:34.100400 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.101915 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.105023 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.105981 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.106322 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:34.107454 (+ 1132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:34.107784 (+ 330us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:34.107941 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:34.108238 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 02:59:34.108470 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260812 02:59:34.110678 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:34.095093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33981 (local address 127.10.118.66:58443)
0812 02:59:34.097946 (+ 2853us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:34.098004 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:34.098332 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:34.099309 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:34.099338 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:34.099391 (+ 53us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:59:34.100096 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:34.100146 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.102183 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.102218 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:34.104722 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:34.104770 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.106364 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.106394 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:34.106937 (+ 543us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:34.107115 (+ 178us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:59:34.107976 (+ 861us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:59:34.108015 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 02:59:34.108153 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 02:59:34.108472 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2285,"thread_start_us":1925,"threads_started":1}
I20260812 02:59:34.223695 13916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:34.204340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:60687 (local address 127.10.118.66:33045)
0812 02:59:34.204916 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:34.204944 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:34.204988 (+ 44us) server_negotiation.cc:407] Connection header received
0812 02:59:34.205153 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:34.205178 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:34.205488 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:34.205651 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:34.213853 (+ 8202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.215146 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.218311 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.219319 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.219556 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:34.220942 (+ 1386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:34.221195 (+ 253us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:34.221315 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:34.221758 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 02:59:34.222002 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 02:59:34.227105 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:34.202398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:33045 (local address 127.10.118.65:60687)
0812 02:59:34.204003 (+ 1605us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:34.204072 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:34.204424 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:34.212533 (+ 8109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:34.212562 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:34.212625 (+ 63us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:59:34.213536 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:34.213578 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.215459 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.215485 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:34.217966 (+ 2481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:34.218016 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.219932 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.219962 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:34.220403 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:34.220592 (+ 189us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:59:34.221447 (+ 855us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:59:34.221479 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 02:59:34.224616 (+ 3137us) client_negotiation.cc:240] Negotiation successful
0812 02:59:34.224959 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":3475,"threads_started":1}
I20260812 02:59:34.351847 12849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:34.352680 12849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:34.362421 12849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5c74b38776942999ad850207e5da639: Sending DeleteTablet for 3 replicas of tablet 118d358deee04ac0bb34a7f98d7da325
I20260812 02:59:34.364987 13663 tablet_service.cc:1514] Processing DeleteTablet for tablet 118d358deee04ac0bb34a7f98d7da325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:34 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:45658
I20260812 02:59:34.366401 13966 tablet_replica.cc:330] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: stopping tablet replica
I20260812 02:59:34.366593 13400 tablet_service.cc:1514] Processing DeleteTablet for tablet 118d358deee04ac0bb34a7f98d7da325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:34 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:44218
I20260812 02:59:34.367470 13966 raft_consensus.cc:2229] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:34.367712 13111 tablet_service.cc:1514] Processing DeleteTablet for tablet 118d358deee04ac0bb34a7f98d7da325 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:34 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:39686
I20260812 02:59:34.368151 13966 raft_consensus.cc:2258] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:34.371245 13966 ts_tablet_manager.cc:1905] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:34.387858 13966 ts_tablet_manager.cc:1918] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:34.388299 13966 log.cc:1196] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/118d358deee04ac0bb34a7f98d7da325
I20260812 02:59:34.389261 13966 ts_tablet_manager.cc:1939] T 118d358deee04ac0bb34a7f98d7da325 P 1ac44449206647aa9cfb0a7f30b4592f: Deleting consensus metadata
I20260812 02:59:34.392001 12824 catalog_manager.cc:4841] TS 1ac44449206647aa9cfb0a7f30b4592f (127.10.118.67:40349): tablet 118d358deee04ac0bb34a7f98d7da325 (table test-table [id=55f8640a60124ff3af41acb1d53c8ba5]) successfully deleted
I20260812 02:59:34.393625 13968 tablet_replica.cc:330] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: stopping tablet replica
I20260812 02:59:34.395464 13967 tablet_replica.cc:330] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: stopping tablet replica
I20260812 02:59:34.396292 13968 raft_consensus.cc:2229] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:34.397202 13967 raft_consensus.cc:2229] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:59:34.397795 13968 raft_consensus.cc:2258] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:34.398751 13967 raft_consensus.cc:2258] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:34.404610 13967 ts_tablet_manager.cc:1905] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:34.405181 13968 ts_tablet_manager.cc:1905] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 02:59:34 dist-test-slave-zpvx krb5kdc[12757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503572, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 02:59:34.434108 13968 ts_tablet_manager.cc:1918] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:34.434514 13968 log.cc:1196] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/118d358deee04ac0bb34a7f98d7da325
I20260812 02:59:34.435289 13968 ts_tablet_manager.cc:1939] T 118d358deee04ac0bb34a7f98d7da325 P 2936a02299544c36b6de6077366dc191: Deleting consensus metadata
I20260812 02:59:34.438292 12818 catalog_manager.cc:4841] TS 2936a02299544c36b6de6077366dc191 (127.10.118.65:33981): tablet 118d358deee04ac0bb34a7f98d7da325 (table test-table [id=55f8640a60124ff3af41acb1d53c8ba5]) successfully deleted
I20260812 02:59:34.443375 13933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:34.395769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.10.118.65:33981)
0812 02:59:34.396324 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:34.396351 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:34.409945 (+ 13594us) server_negotiation.cc:407] Connection header received
0812 02:59:34.410347 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:34.410375 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:34.410628 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:34.410820 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:34.413564 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.415190 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.418181 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.419206 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.419468 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:34.428225 (+ 8757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:34.428267 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:34.428312 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:34.428489 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:34.435181 (+ 6692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:34.436836 (+ 1655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:34.436869 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:34.436892 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:34.437150 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:34.438055 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:34.438130 (+ 75us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:34.438173 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:34.439042 (+ 869us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:34.439567 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:34.440805 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0812 02:59:34.441280 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260812 02:59:34.443472 13967 ts_tablet_manager.cc:1918] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:34.443992 13967 log.cc:1196] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/118d358deee04ac0bb34a7f98d7da325
I20260812 02:59:34.444818 13967 ts_tablet_manager.cc:1939] T 118d358deee04ac0bb34a7f98d7da325 P fb55a80c5d094ded836ac0eafd57c18d: Deleting consensus metadata
W20260812 02:59:34.446321 13227 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39706
I20260812 02:59:34.448014 12818 catalog_manager.cc:4841] TS fb55a80c5d094ded836ac0eafd57c18d (127.10.118.66:33045): tablet 118d358deee04ac0bb34a7f98d7da325 (table test-table [id=55f8640a60124ff3af41acb1d53c8ba5]) successfully deleted
I20260812 02:59:34.523782 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:34.457125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.10.118.126:40205)
0812 02:59:34.479015 (+ 21890us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:34.479036 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:34.479074 (+ 38us) server_negotiation.cc:407] Connection header received
0812 02:59:34.479195 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:34.479211 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:34.479385 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:34.479589 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:34.482395 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.483870 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.486457 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:34.487397 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:34.487676 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:34.509554 (+ 21878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:34.509588 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:34.509624 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:34.509757 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:34.515048 (+ 5291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:34.516581 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:34.516620 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:34.516641 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:34.516889 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:34.517695 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:34.517724 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:34.517748 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:34.518414 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:34.518801 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:34.521133 (+ 2332us) server_negotiation.cc:299] Negotiation successful
0812 02:59:34.521608 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21580,"thread_start_us":23184,"threads_started":1}
W20260812 02:59:34.529376 12849 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36392
I20260812 02:59:34.532665 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 12971
I20260812 02:59:34.619412 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 13250
I20260812 02:59:34.665524 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 13529
I20260812 02:59:34.698438 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 12780
2026-08-12T02:59:34Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (11845 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:59:34 dist-test-slave-zpvx krb5kdc[13982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:59:34 dist-test-slave-zpvx krb5kdc[13982](info): set up 2 sockets
Aug 12 02:59:34 dist-test-slave-zpvx krb5kdc[13982](info): commencing operation
krb5kdc: starting...
W20260812 02:59:36.092391 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.319s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 02:59:36 dist-test-slave-zpvx krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:59:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:59:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 02:59:36.302521 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:42029
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:42029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:36.609122 14001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:36.609741 14001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:36.610349 14001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:36.641238 14001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:59:36.641516 14001 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 02:59:36.641834 14001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:36.642140 14001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:36.642393 14001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:59:36.642644 14001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:59:36.678758 14001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:42029
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:42029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:36.679986 14001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:36.681161 14001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:36.700759 14016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:36.700834 14020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:36.701090 14017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:36.703307 14001 server_base.cc:1034] running on GCE node
I20260812 02:59:37.937258 14001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:37.939832 14001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:37.941185 14001 hybrid_clock.cc:648] HybridClock initialized: now 1786503577941142 us; error 66 us; skew 500 ppm
I20260812 02:59:37.942006 14001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:37 dist-test-slave-zpvx krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503577, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:37.948012 14001 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 02:59:37.953431 14001 webserver.cc:460] Webserver started at http://127.10.118.126:34499/ using document root <none> and password file <none>
I20260812 02:59:37.954337 14001 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:37.954522 14001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:37.954929 14001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:37.960300 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "bf726659b71642368fff36ad5fbafeba"
format_stamp: "Formatted at 2026-08-12 02:59:37 on dist-test-slave-zpvx"
server_key: "a952ea19ffda038771d1c346a7ecc601"
server_key_iv: "473461684b002cc209018cb1408c5236"
server_key_version: "encryptionkey@0"
I20260812 02:59:37.961476 14001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "bf726659b71642368fff36ad5fbafeba"
format_stamp: "Formatted at 2026-08-12 02:59:37 on dist-test-slave-zpvx"
server_key: "a952ea19ffda038771d1c346a7ecc601"
server_key_iv: "473461684b002cc209018cb1408c5236"
server_key_version: "encryptionkey@0"
I20260812 02:59:37.968485 14001 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 02:59:37.974007 14040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:37.975078 14001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:59:37.975442 14001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "bf726659b71642368fff36ad5fbafeba"
format_stamp: "Formatted at 2026-08-12 02:59:37 on dist-test-slave-zpvx"
server_key: "a952ea19ffda038771d1c346a7ecc601"
server_key_iv: "473461684b002cc209018cb1408c5236"
server_key_version: "encryptionkey@0"
I20260812 02:59:37.975802 14001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:38.016732 14001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:38.022176 14001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:38.022665 14001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:38.103863 14001 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:42029
I20260812 02:59:38.103982 14149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:42029 every 8 connection(s)
I20260812 02:59:38.106654 14001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 02:59:38.111825 14153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:38.116012 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 14001
I20260812 02:59:38.116523 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 02:59:38.117624 10713 external_mini_cluster.cc:1428] Setting key 8378c033d5f029ad5bfbe96c8dc6ec2b
I20260812 02:59:38.135824 14153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Bootstrap starting.
Aug 12 02:59:38 dist-test-slave-zpvx krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:38.141520 14153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:38.143591 14153 log.cc:824] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:38.149358 14153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: No bootstrap required, opened a new log
I20260812 02:59:38.155016 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:38.119932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.10.118.126:42029)
0812 02:59:38.122498 (+ 2566us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:38.122535 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:38.122611 (+ 76us) server_negotiation.cc:407] Connection header received
0812 02:59:38.128192 (+ 5581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:38.128251 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:38.128750 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:38.129215 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:38.131373 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:38.134037 (+ 2664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:38.135813 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:38.137196 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:38.137535 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:38.142646 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:38.142681 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:38.142724 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:38.142876 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:38.148584 (+ 5708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:38.149720 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:38.149743 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:38.149760 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:38.150005 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:38.150633 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:38.150657 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:38.150670 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:38.151371 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:38.151843 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:38.152568 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 02:59:38.153177 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2031,"thread_start_us":1624,"threads_started":1}
I20260812 02:59:38.171384 14153 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf726659b71642368fff36ad5fbafeba" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42029 } }
I20260812 02:59:38.172152 14153 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:38.172415 14153 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf726659b71642368fff36ad5fbafeba, State: Initialized, Role: FOLLOWER
I20260812 02:59:38.173101 14153 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf726659b71642368fff36ad5fbafeba" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42029 } }
I20260812 02:59:38.173641 14153 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:59:38.173921 14153 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:59:38.174297 14153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:38.179016 14153 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf726659b71642368fff36ad5fbafeba" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42029 } }
I20260812 02:59:38.179754 14153 leader_election.cc:302] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf726659b71642368fff36ad5fbafeba; no voters:
I20260812 02:59:38.181483 14153 leader_election.cc:288] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:59:38.181881 14162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:38.185197 14162 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [term 1 LEADER]: Becoming Leader. State: Replica: bf726659b71642368fff36ad5fbafeba, State: Running, Role: LEADER
I20260812 02:59:38.185930 14153 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:59:38.185925 14162 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf726659b71642368fff36ad5fbafeba" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42029 } }
I20260812 02:59:38.197538 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf726659b71642368fff36ad5fbafeba. Latest consensus state: current_term: 1 leader_uuid: "bf726659b71642368fff36ad5fbafeba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf726659b71642368fff36ad5fbafeba" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42029 } } }
I20260812 02:59:38.198433 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:38.200662 14165 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf726659b71642368fff36ad5fbafeba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf726659b71642368fff36ad5fbafeba" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42029 } } }
I20260812 02:59:38.201334 14165 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:38.204587 14179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:59:38.215183 14179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:59:38.232534 14179 catalog_manager.cc:1347] Generated new cluster ID: 3654008539b84bc896c0c931d34a3f9f
I20260812 02:59:38.232839 14179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:59:38.254057 14179 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:59:38.255785 14179 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:59:38.271528 14179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Generated new TSK 0
I20260812 02:59:38.272472 14179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 02:59:38.329110 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42029
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:38.635921 14205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:38.636480 14205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:38.637207 14205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:38.668592 14205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:38.668957 14205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:38.669629 14205 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:59:38.669857 14205 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:59:38.670356 14205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 02:59:38.710976 14205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:42029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:38.712143 14205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:38.713179 14205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:38.734025 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:38.734117 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:38.734953 14219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:38.736274 14205 server_base.cc:1034] running on GCE node
I20260812 02:59:39.914193 14205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:39.920773 14205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:39.922168 14205 hybrid_clock.cc:648] HybridClock initialized: now 1786503579922135 us; error 71 us; skew 500 ppm
I20260812 02:59:39.922994 14205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:39 dist-test-slave-zpvx krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503579, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:39.929214 14205 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 02:59:39.934625 14205 webserver.cc:460] Webserver started at http://127.10.118.65:39741/ using document root <none> and password file <none>
I20260812 02:59:39.935575 14205 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:39.935828 14205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:39.936288 14205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:39.941704 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "fb14d2640cf6496ebeea2f59e42cc484"
format_stamp: "Formatted at 2026-08-12 02:59:39 on dist-test-slave-zpvx"
server_key: "0bb73ee301fb28b287d449ee30f66e28"
server_key_iv: "bee35ea56a499f89587a56b64d89a306"
server_key_version: "encryptionkey@0"
I20260812 02:59:39.942876 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "fb14d2640cf6496ebeea2f59e42cc484"
format_stamp: "Formatted at 2026-08-12 02:59:39 on dist-test-slave-zpvx"
server_key: "0bb73ee301fb28b287d449ee30f66e28"
server_key_iv: "bee35ea56a499f89587a56b64d89a306"
server_key_version: "encryptionkey@0"
I20260812 02:59:39.950482 14205 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.007s
I20260812 02:59:39.956058 14235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:39.957140 14205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 02:59:39.957489 14205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "fb14d2640cf6496ebeea2f59e42cc484"
format_stamp: "Formatted at 2026-08-12 02:59:39 on dist-test-slave-zpvx"
server_key: "0bb73ee301fb28b287d449ee30f66e28"
server_key_iv: "bee35ea56a499f89587a56b64d89a306"
server_key_version: "encryptionkey@0"
I20260812 02:59:39.957885 14205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:40.004563 14205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:40.010411 14205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:40.010907 14205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:40.019970 14205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:40.020404 14205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:40.020749 14205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:40.020984 14205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:59:40 dist-test-slave-zpvx krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503579, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:40.056559 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:40.017560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:53513 (local address 127.10.118.126:42029)
0812 02:59:40.019053 (+ 1493us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:40.019076 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:40.021195 (+ 2119us) server_negotiation.cc:407] Connection header received
0812 02:59:40.028931 (+ 7736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:40.028951 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:40.029134 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:40.029289 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:40.032646 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.034104 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.037951 (+ 3847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.038876 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.039172 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:40.045645 (+ 6473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:40.045670 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:40.045686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:40.045787 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:40.049634 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:40.050828 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:40.050848 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:40.050864 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:40.051051 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:40.051790 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:40.051809 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:40.051822 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:40.052315 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:40.052619 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:40.053736 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0812 02:59:40.054206 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":612,"threads_started":1}
I20260812 02:59:40.057405 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:40.017669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42029 (local address 127.10.118.65:53513)
0812 02:59:40.020908 (+ 3239us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:40.021004 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:40.028543 (+ 7539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:40.029616 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:40.029655 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:40.030894 (+ 1239us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:40.032181 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:40.032257 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.034437 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.034496 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:40.037601 (+ 3105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:40.037669 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.039211 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.039242 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:40.039716 (+ 474us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:40.040731 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:40.040851 (+ 120us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:40.045290 (+ 4439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:40.049855 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:40.049895 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:40.049918 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:40.050608 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:40.051256 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:40.051286 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:40.051300 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:40.051577 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:40.052579 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:40.052633 (+ 54us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:40.053108 (+ 475us) client_negotiation.cc:769] Sending connection context
0812 02:59:40.053493 (+ 385us) client_negotiation.cc:240] Negotiation successful
0812 02:59:40.054257 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2671,"thread_start_us":2279,"threads_started":1}
I20260812 02:59:40.102095 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:40.073079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:48427 (local address 127.10.118.126:42029)
0812 02:59:40.073651 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:40.073683 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:40.076774 (+ 3091us) server_negotiation.cc:407] Connection header received
0812 02:59:40.077159 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:40.077178 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:40.077347 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:40.077515 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:40.080181 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.081410 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.083396 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.083989 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.084274 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:40.088061 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:40.088091 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:40.088120 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:40.088276 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:40.094420 (+ 6144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:40.096118 (+ 1698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:40.096163 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:40.096202 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:40.096529 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:40.097670 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:40.097700 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:40.097724 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:40.098320 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:40.098575 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:40.099380 (+ 805us) server_negotiation.cc:299] Negotiation successful
0812 02:59:40.099861 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260812 02:59:40.102792 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:40.073009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42029 (local address 127.10.118.65:48427)
0812 02:59:40.076561 (+ 3552us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:40.076653 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:40.076938 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:40.077672 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:40.077698 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:40.078741 (+ 1043us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:40.079840 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:40.079901 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.081736 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.081755 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:40.083137 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:40.083174 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:40.084298 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:40.084329 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:40.084728 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:40.085536 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:40.085596 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:40.087629 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:40.094685 (+ 7056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:40.094724 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:40.094751 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:40.095862 (+ 1111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:40.096809 (+ 947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:40.096858 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:40.096892 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:40.097259 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:40.098539 (+ 1280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:40.098565 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:40.098891 (+ 326us) client_negotiation.cc:769] Sending connection context
0812 02:59:40.099161 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 02:59:40.099779 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3166,"thread_start_us":2030,"threads_started":1}
I20260812 02:59:40.189353 14205 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:35707
I20260812 02:59:40.189947 14496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:35707 every 8 connection(s)
I20260812 02:59:40.191865 14205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 02:59:40.192884 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 14205
I20260812 02:59:40.193473 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 02:59:40.195135 10713 external_mini_cluster.cc:1428] Setting key 219d14c92bd10298adfe63c41adc4402
I20260812 02:59:40.197221 14500 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42029
I20260812 02:59:40.197657 14500 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:40.198767 14500 heartbeater.cc:511] Master 127.10.118.126:42029 requested a full tablet report, sending...
I20260812 02:59:40.201717 14078 ts_manager.cc:194] Registered new tserver with Master: fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707)
I20260812 02:59:40.204874 14078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:53513
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 02:59:40.250272 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42029
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:40.554785 14504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:40.555267 14504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:40.555907 14504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:40.586449 14504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:40.586830 14504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:40.587447 14504 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:59:40.587617 14504 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:59:40.588130 14504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 02:59:40.622959 14504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:42029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:40.624269 14504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:40.625257 14504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:40.643397 14516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:41.208932 14500 heartbeater.cc:503] Master 127.10.118.126:42029 was elected leader, sending a full tablet report...
W20260812 02:59:40.643440 14518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:40.644196 14522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:40.644971 14504 server_base.cc:1034] running on GCE node
I20260812 02:59:41.829200 14504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:41.831518 14504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:41.832858 14504 hybrid_clock.cc:648] HybridClock initialized: now 1786503581832828 us; error 45 us; skew 500 ppm
I20260812 02:59:41.833638 14504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:41 dist-test-slave-zpvx krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503581, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:41.839511 14504 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 02:59:41.844849 14504 webserver.cc:460] Webserver started at http://127.10.118.66:39293/ using document root <none> and password file <none>
I20260812 02:59:41.845763 14504 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:41.845975 14504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:41.846470 14504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:41.852263 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "395f2335bee84faa89d4121babfc77d9"
format_stamp: "Formatted at 2026-08-12 02:59:41 on dist-test-slave-zpvx"
server_key: "86bb4b7b285138adee39cf8f19f3e6f1"
server_key_iv: "4a4630cf6bc6ae9c33deb5f3ec6fc5d8"
server_key_version: "encryptionkey@0"
I20260812 02:59:41.853518 14504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "395f2335bee84faa89d4121babfc77d9"
format_stamp: "Formatted at 2026-08-12 02:59:41 on dist-test-slave-zpvx"
server_key: "86bb4b7b285138adee39cf8f19f3e6f1"
server_key_iv: "4a4630cf6bc6ae9c33deb5f3ec6fc5d8"
server_key_version: "encryptionkey@0"
I20260812 02:59:41.860982 14504 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 02:59:41.866552 14537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:41.867698 14504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260812 02:59:41.868033 14504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "395f2335bee84faa89d4121babfc77d9"
format_stamp: "Formatted at 2026-08-12 02:59:41 on dist-test-slave-zpvx"
server_key: "86bb4b7b285138adee39cf8f19f3e6f1"
server_key_iv: "4a4630cf6bc6ae9c33deb5f3ec6fc5d8"
server_key_version: "encryptionkey@0"
I20260812 02:59:41.868440 14504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:41.908871 14504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:41.914726 14504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:41.915191 14504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:41.925154 14504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:41.925526 14504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 02:59:41.925909 14504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:41.926200 14504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:59:41 dist-test-slave-zpvx krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503581, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:41.962474 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:41.922211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:36341 (local address 127.10.118.126:42029)
0812 02:59:41.925749 (+ 3538us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:41.925771 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:41.928094 (+ 2323us) server_negotiation.cc:407] Connection header received
0812 02:59:41.935729 (+ 7635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:41.935751 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:41.935938 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:41.936097 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:41.939688 (+ 3591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.940726 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.944469 (+ 3743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.945473 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.945776 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:41.951111 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:41.951135 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:41.951151 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:41.951250 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:41.954998 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:41.956219 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:41.956239 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:41.956253 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:41.956512 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:41.957230 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:41.957249 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:41.957262 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:41.957755 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:41.957997 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:41.959130 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0812 02:59:41.959777 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3095,"thread_start_us":1787,"threads_started":1}
I20260812 02:59:41.963203 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:41.922453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42029 (local address 127.10.118.66:36341)
0812 02:59:41.927744 (+ 5291us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:41.927861 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:41.935327 (+ 7466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:41.936467 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:41.936507 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:41.937773 (+ 1266us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:41.939281 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:41.939380 (+ 99us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.940987 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.941023 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:41.944094 (+ 3071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:41.944147 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.945837 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.945873 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:41.946339 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:41.947240 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:41.947310 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:41.950761 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:41.955227 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:41.955271 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:41.955294 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:41.956001 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:41.956704 (+ 703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:41.956738 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:41.956752 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:41.957022 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:41.957977 (+ 955us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:41.958030 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:41.958471 (+ 441us) client_negotiation.cc:769] Sending connection context
0812 02:59:41.958897 (+ 426us) client_negotiation.cc:240] Negotiation successful
0812 02:59:41.959671 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4611,"thread_start_us":3450,"threads_started":1}
I20260812 02:59:42.011214 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:41.980323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42029 (local address 127.10.118.66:45149)
0812 02:59:41.983965 (+ 3642us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:41.984016 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:41.984343 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:41.985676 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:41.985708 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:41.987015 (+ 1307us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:41.987990 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:41.988052 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.990205 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.990245 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:41.992891 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:41.992955 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.994669 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.994712 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:41.995264 (+ 552us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:41.996398 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:41.996517 (+ 119us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:41.998597 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:42.004354 (+ 5757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:42.004407 (+ 53us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:42.004432 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:42.005013 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:42.005652 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:42.005679 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:42.005708 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:42.005974 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:42.007570 (+ 1596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:42.007608 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:42.008120 (+ 512us) client_negotiation.cc:769] Sending connection context
0812 02:59:42.008551 (+ 431us) client_negotiation.cc:240] Negotiation successful
0812 02:59:42.009240 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3312,"thread_start_us":2093,"threads_started":1}
I20260812 02:59:42.012068 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:41.980394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:45149 (local address 127.10.118.126:42029)
0812 02:59:41.980990 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:41.981022 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:41.984155 (+ 3133us) server_negotiation.cc:407] Connection header received
0812 02:59:41.984671 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:41.984712 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:41.985033 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:41.985250 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:41.988322 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.989971 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.993197 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:41.994314 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:41.994669 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:41.999017 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:41.999061 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:41.999102 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:41.999282 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:42.004123 (+ 4841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:42.005233 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:42.005254 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:42.005268 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:42.005458 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:42.006262 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:42.006300 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:42.006335 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:42.007182 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:42.007544 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:42.008868 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0812 02:59:42.009426 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260812 02:59:42.093329 14504 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:38547
I20260812 02:59:42.093976 14773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:38547 every 8 connection(s)
I20260812 02:59:42.095916 14504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 02:59:42.100548 14774 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42029
I20260812 02:59:42.101001 14774 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:42.102037 14774 heartbeater.cc:511] Master 127.10.118.126:42029 requested a full tablet report, sending...
I20260812 02:59:42.104126 14078 ts_manager.cc:194] Registered new tserver with Master: 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547)
I20260812 02:59:42.105042 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 14504
I20260812 02:59:42.105630 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 02:59:42.106304 14078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:36341
I20260812 02:59:42.106937 10713 external_mini_cluster.cc:1428] Setting key ac916151027b1287c413e5a533d9ccdb
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 02:59:42.160977 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42029
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:42.469552 14787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:42.470310 14787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:42.471138 14787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:42.504935 14787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:42.505313 14787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:42.505970 14787 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 02:59:42.506223 14787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 02:59:42.506775 14787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 02:59:42.542332 14787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:42029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:42.543519 14787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:42.544620 14787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:42.564515 14804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:43.110605 14774 heartbeater.cc:503] Master 127.10.118.126:42029 was elected leader, sending a full tablet report...
I20260812 02:59:42.566042 14787 server_base.cc:1034] running on GCE node
W20260812 02:59:42.564790 14801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:42.564560 14802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:43.756480 14787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:43.759016 14787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:43.760509 14787 hybrid_clock.cc:648] HybridClock initialized: now 1786503583760491 us; error 66 us; skew 500 ppm
I20260812 02:59:43.761422 14787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:43 dist-test-slave-zpvx krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503583, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:43.773926 14787 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 02:59:43.779486 14787 webserver.cc:460] Webserver started at http://127.10.118.67:39255/ using document root <none> and password file <none>
I20260812 02:59:43.780620 14787 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:43.780879 14787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:43.781463 14787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:43.788300 14787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "9e1ab11b7f874fabb66918d268d271fa"
format_stamp: "Formatted at 2026-08-12 02:59:43 on dist-test-slave-zpvx"
server_key: "79e4aa4c1d2abf281263a18ded450544"
server_key_iv: "9ad1b94243f95488752e552a36af5e91"
server_key_version: "encryptionkey@0"
I20260812 02:59:43.789863 14787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "9e1ab11b7f874fabb66918d268d271fa"
format_stamp: "Formatted at 2026-08-12 02:59:43 on dist-test-slave-zpvx"
server_key: "79e4aa4c1d2abf281263a18ded450544"
server_key_iv: "9ad1b94243f95488752e552a36af5e91"
server_key_version: "encryptionkey@0"
I20260812 02:59:43.797904 14787 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 02:59:43.803735 14817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:43.804859 14787 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 02:59:43.805236 14787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "9e1ab11b7f874fabb66918d268d271fa"
format_stamp: "Formatted at 2026-08-12 02:59:43 on dist-test-slave-zpvx"
server_key: "79e4aa4c1d2abf281263a18ded450544"
server_key_iv: "9ad1b94243f95488752e552a36af5e91"
server_key_version: "encryptionkey@0"
I20260812 02:59:43.805635 14787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:43.852154 14787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:43.858032 14787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:43.858529 14787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:43.869771 14787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:43.870122 14787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:43.870435 14787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:43.870618 14787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 02:59:43 dist-test-slave-zpvx krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503583, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:43.906353 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:43.865811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:33627 (local address 127.10.118.126:42029)
0812 02:59:43.867056 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:43.867079 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:43.870673 (+ 3594us) server_negotiation.cc:407] Connection header received
0812 02:59:43.875973 (+ 5300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:43.876010 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:43.876284 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:43.876530 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:43.879067 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.880038 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.882637 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.883639 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.883951 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:43.891670 (+ 7719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:43.891697 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:43.891713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:43.891814 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:43.897432 (+ 5618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:43.899485 (+ 2053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:43.899531 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:43.899570 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:43.899883 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:43.900986 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:43.901029 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:43.901063 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:43.901852 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:43.902150 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:43.903884 (+ 1734us) server_negotiation.cc:299] Negotiation successful
0812 02:59:43.904513 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":673,"threads_started":1}
I20260812 02:59:43.909072 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:43.866140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42029 (local address 127.10.118.67:33627)
0812 02:59:43.870391 (+ 4251us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:43.870450 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:43.875576 (+ 5126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:43.876849 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:43.876874 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:43.877784 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:43.878756 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:43.878802 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.880311 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.880332 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:43.882239 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:43.882308 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.884092 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.884143 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:43.884898 (+ 755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:43.886102 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:43.886166 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:43.891247 (+ 5081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:43.897725 (+ 6478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:43.897770 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:43.897824 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:43.899167 (+ 1343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:43.900130 (+ 963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:43.900161 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:43.900194 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:43.900684 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:43.902119 (+ 1435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:43.902183 (+ 64us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:43.902998 (+ 815us) client_negotiation.cc:769] Sending connection context
0812 02:59:43.903539 (+ 541us) client_negotiation.cc:240] Negotiation successful
0812 02:59:43.904683 (+ 1144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3744,"thread_start_us":2661,"threads_started":1}
I20260812 02:59:43.960994 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:43.925961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:54259 (local address 127.10.118.126:42029)
0812 02:59:43.926617 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:43.926649 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:43.929326 (+ 2677us) server_negotiation.cc:407] Connection header received
0812 02:59:43.929683 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:43.929717 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:43.930006 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:43.930216 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:43.933738 (+ 3522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.935580 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.939112 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.940319 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.940781 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:43.945250 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:43.945290 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:43.945322 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:43.945486 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:43.952073 (+ 6587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:43.953913 (+ 1840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:43.953961 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:43.953991 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:43.954263 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:43.955264 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:43.955306 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:43.955334 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:43.956117 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:43.956432 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:43.957604 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0812 02:59:43.958213 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260812 02:59:43.961419 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:43.925813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42029 (local address 127.10.118.67:54259)
0812 02:59:43.929117 (+ 3304us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:43.929178 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:43.929420 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:43.930786 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:43.930829 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:43.932221 (+ 1392us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:43.933344 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:43.933412 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.935916 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.935956 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:43.938738 (+ 2782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:43.938812 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:43.940729 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:43.940823 (+ 94us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:43.941380 (+ 557us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:43.942549 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:43.942658 (+ 109us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:43.944802 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:43.952387 (+ 7585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:43.952431 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:43.952476 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:43.953552 (+ 1076us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:43.954511 (+ 959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:43.954555 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:43.954591 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:43.954984 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:43.956449 (+ 1465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:43.956502 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:43.956947 (+ 445us) client_negotiation.cc:769] Sending connection context
0812 02:59:43.957304 (+ 357us) client_negotiation.cc:240] Negotiation successful
0812 02:59:43.958066 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2921,"thread_start_us":2487,"threads_started":1}
I20260812 02:59:44.037047 14787 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:36403
I20260812 02:59:44.037639 15047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:36403 every 8 connection(s)
I20260812 02:59:44.039603 14787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 02:59:44.040913 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 14787
I20260812 02:59:44.041756 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 02:59:44.043471 10713 external_mini_cluster.cc:1428] Setting key 53ce80663700950238498ba7c76f2f6e
I20260812 02:59:44.045063 15048 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42029
I20260812 02:59:44.045418 15048 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:44.046514 15048 heartbeater.cc:511] Master 127.10.118.126:42029 requested a full tablet report, sending...
I20260812 02:59:44.049213 14077 ts_manager.cc:194] Registered new tserver with Master: 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403)
I20260812 02:59:44.050747 14077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:33627
I20260812 02:59:44.063261 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 02:59:44 dist-test-slave-zpvx krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 02:59:44 dist-test-slave-zpvx krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:44.143955 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.117984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.10.118.126:42029)
0812 02:59:44.118493 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:44.118517 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:44.119850 (+ 1333us) server_negotiation.cc:407] Connection header received
0812 02:59:44.120272 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:44.120291 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:44.120526 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:44.120657 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:44.123975 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.124984 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.127670 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.128476 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.128759 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.134936 (+ 6177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:44.134956 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:44.134975 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:44.135073 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:44.138424 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:44.139489 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:44.139511 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:44.139524 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:44.139724 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:44.140453 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:44.140474 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:44.140491 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:44.140985 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:44.141365 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:44.142054 (+ 689us) server_negotiation.cc:299] Negotiation successful
0812 02:59:44.142514 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260812 02:59:44.150171 14077 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:59:44.171147 14077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:44.225193 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.200126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.0.0.1:36892)
0812 02:59:44.203915 (+ 3789us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:44.203997 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:44.204349 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:44.211815 (+ 7466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:44.211838 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:44.211962 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:44.212547 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.212603 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.216017 (+ 3414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.216052 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.219250 (+ 3198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.219306 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.221970 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.222000 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.222515 (+ 515us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.222699 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 02:59:44.222918 (+ 219us) client_negotiation.cc:240] Negotiation successful
0812 02:59:44.223313 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3278,"mutex_wait_us":155,"thread_start_us":2974,"threads_started":1}
I20260812 02:59:44.226102 15070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.200698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.10.118.65:35707)
0812 02:59:44.210587 (+ 9889us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:44.210638 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:44.210711 (+ 73us) server_negotiation.cc:407] Connection header received
0812 02:59:44.210944 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:44.210991 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:44.211432 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:44.211717 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:44.212894 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.215698 (+ 2804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.219563 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.221588 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.223595 (+ 2007us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.224038 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:44.224195 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 02:59:44.224516 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9265,"thread_start_us":8864,"threads_started":1}
I20260812 02:59:44.228585 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.201934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:36403 (local address 127.0.0.1:53524)
0812 02:59:44.206744 (+ 4810us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:44.206999 (+ 255us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:44.207270 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:44.213458 (+ 6188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:44.213486 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:44.213583 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:44.214328 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.214379 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.217080 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.217101 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.219513 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.219569 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.224864 (+ 5295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.224900 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.225353 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.225514 (+ 161us) client_negotiation.cc:769] Sending connection context
0812 02:59:44.225632 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 02:59:44.225934 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4431,"mutex_wait_us":113,"thread_start_us":4074,"threads_started":1}
I20260812 02:59:44.229153 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.207757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.10.118.66:38547)
0812 02:59:44.215190 (+ 7433us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:44.215222 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:44.215272 (+ 50us) server_negotiation.cc:407] Connection header received
0812 02:59:44.215438 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:44.215468 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:44.215807 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:44.216017 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:44.218060 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.219778 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.223125 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.225014 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.225335 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.225756 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:44.226966 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0812 02:59:44.227422 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6998,"thread_start_us":6577,"threads_started":1}
I20260812 02:59:44.229079 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.207157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:38547 (local address 127.0.0.1:57512)
0812 02:59:44.209588 (+ 2431us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:44.209647 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:44.209892 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:44.216546 (+ 6654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:44.216577 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:44.216688 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:44.217730 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.217795 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.220028 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.220049 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.222751 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.222804 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.226023 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.226058 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.226499 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.226645 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 02:59:44.226788 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 02:59:44.227082 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1997,"mutex_wait_us":73,"thread_start_us":1663,"threads_started":1}
I20260812 02:59:44.230664 14354 tablet_service.cc:1467] Processing CreateTablet for tablet 702770a8d26b4b269fb3f6cc5ee1d217 (DEFAULT_TABLE table=test-table [id=1352140ee24d45ef8c735037d221c672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:44.232136 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.203349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.10.118.67:36403)
0812 02:59:44.206394 (+ 3045us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:44.206426 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:44.212319 (+ 5893us) server_negotiation.cc:407] Connection header received
0812 02:59:44.212629 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:44.212681 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:44.213127 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:44.213395 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:44.214741 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.216835 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.219891 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.224466 (+ 4575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.230154 (+ 5688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.230591 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:44.230752 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 02:59:44.230955 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2498,"thread_start_us":2247,"threads_started":1}
I20260812 02:59:44.232605 14354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702770a8d26b4b269fb3f6cc5ee1d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:44.233603 14631 tablet_service.cc:1467] Processing CreateTablet for tablet 702770a8d26b4b269fb3f6cc5ee1d217 (DEFAULT_TABLE table=test-table [id=1352140ee24d45ef8c735037d221c672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:44.235837 14631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702770a8d26b4b269fb3f6cc5ee1d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:44.235945 14919 tablet_service.cc:1467] Processing CreateTablet for tablet 702770a8d26b4b269fb3f6cc5ee1d217 (DEFAULT_TABLE table=test-table [id=1352140ee24d45ef8c735037d221c672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:44.237531 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 702770a8d26b4b269fb3f6cc5ee1d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:44.259538 15082 tablet_bootstrap.cc:492] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Bootstrap starting.
I20260812 02:59:44.262264 15086 tablet_bootstrap.cc:492] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Bootstrap starting.
I20260812 02:59:44.267066 15087 tablet_bootstrap.cc:492] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Bootstrap starting.
I20260812 02:59:44.269235 15082 tablet_bootstrap.cc:654] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:44.269420 15086 tablet_bootstrap.cc:654] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:44.271661 15082 log.cc:824] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:44.271656 15086 log.cc:824] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:44.272151 15087 tablet_bootstrap.cc:654] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:44.274611 15087 log.cc:824] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:44.277899 15086 tablet_bootstrap.cc:492] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: No bootstrap required, opened a new log
I20260812 02:59:44.278506 15086 ts_tablet_manager.cc:1397] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260812 02:59:44.279134 15082 tablet_bootstrap.cc:492] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: No bootstrap required, opened a new log
I20260812 02:59:44.279557 15082 ts_tablet_manager.cc:1397] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.010s
I20260812 02:59:44.281324 15087 tablet_bootstrap.cc:492] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: No bootstrap required, opened a new log
I20260812 02:59:44.281774 15087 ts_tablet_manager.cc:1397] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260812 02:59:44.300881 15082 raft_consensus.cc:348] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.301558 15082 raft_consensus.cc:374] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:44.302126 15082 raft_consensus.cc:729] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1ab11b7f874fabb66918d268d271fa, State: Initialized, Role: FOLLOWER
I20260812 02:59:44.302922 15082 consensus_queue.cc:260] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.305817 15048 heartbeater.cc:503] Master 127.10.118.126:42029 was elected leader, sending a full tablet report...
I20260812 02:59:44.306522 15082 ts_tablet_manager.cc:1428] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Time spent starting tablet: real 0.027s user 0.015s sys 0.010s
I20260812 02:59:44.309401 15086 raft_consensus.cc:348] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.310333 15086 raft_consensus.cc:374] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:44.310636 15086 raft_consensus.cc:729] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Initialized, Role: FOLLOWER
I20260812 02:59:44.311496 15087 raft_consensus.cc:348] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.311596 15086 consensus_queue.cc:260] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.312215 15087 raft_consensus.cc:374] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:44.312518 15087 raft_consensus.cc:729] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f2335bee84faa89d4121babfc77d9, State: Initialized, Role: FOLLOWER
I20260812 02:59:44.315157 15086 ts_tablet_manager.cc:1428] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Time spent starting tablet: real 0.036s user 0.022s sys 0.012s
I20260812 02:59:44.313540 15087 consensus_queue.cc:260] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.318748 15087 ts_tablet_manager.cc:1428] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Time spent starting tablet: real 0.037s user 0.032s sys 0.001s
W20260812 02:59:44.352517 14777 tablet.cc:2369] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:59:44.447443 15102 raft_consensus.cc:482] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:44.447917 15102 raft_consensus.cc:504] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.450309 15102 leader_election.cc:288] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403), 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547)
W20260812 02:59:44.451531 14501 tablet.cc:2369] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 02:59:44.465730 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.450933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:34531 (local address 127.10.118.67:36403)
0812 02:59:44.451544 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:44.451574 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:44.453738 (+ 2164us) server_negotiation.cc:407] Connection header received
0812 02:59:44.454384 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:44.454413 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:44.454646 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:44.454828 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:44.455843 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.457450 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.461105 (+ 3655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.462775 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.463121 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.463396 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:44.464015 (+ 619us) server_negotiation.cc:299] Negotiation successful
0812 02:59:44.464279 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 02:59:44.466287 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.450852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:36403 (local address 127.10.118.65:34531)
0812 02:59:44.453553 (+ 2701us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:44.453612 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:44.454000 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:44.454880 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:44.454909 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:44.455009 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:44.455560 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.455596 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.457728 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.457758 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.460861 (+ 3103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.460901 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.463086 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.463117 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.463699 (+ 582us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.463795 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 02:59:44.463960 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 02:59:44.464276 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2238,"mutex_wait_us":282,"thread_start_us":1920,"threads_started":1}
I20260812 02:59:44.467893 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.453973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:45947 (local address 127.10.118.66:38547)
0812 02:59:44.454450 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:44.454477 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:44.456518 (+ 2041us) server_negotiation.cc:407] Connection header received
0812 02:59:44.456973 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:44.457000 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:44.457246 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:44.457550 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:44.459169 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.460447 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.463868 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.464941 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.465306 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.465625 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:44.466308 (+ 683us) server_negotiation.cc:299] Negotiation successful
0812 02:59:44.466597 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 02:59:44.467831 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702770a8d26b4b269fb3f6cc5ee1d217" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1ab11b7f874fabb66918d268d271fa" is_pre_election: true
I20260812 02:59:44.468583 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.453767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:38547 (local address 127.10.118.65:45947)
0812 02:59:44.456293 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:44.456351 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:44.456655 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:44.457665 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:44.457694 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:44.457796 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:44.458807 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.458867 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.460717 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.460752 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.463609 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.463644 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.465284 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.465317 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.465735 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.465830 (+ 95us) client_negotiation.cc:769] Sending connection context
0812 02:59:44.466881 (+ 1051us) client_negotiation.cc:240] Negotiation successful
0812 02:59:44.467073 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2021,"mutex_wait_us":110,"thread_start_us":1696,"threads_started":1}
I20260812 02:59:44.468742 14952 raft_consensus.cc:2454] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 0.
I20260812 02:59:44.469950 14240 leader_election.cc:302] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:44.470175 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702770a8d26b4b269fb3f6cc5ee1d217" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9" is_pre_election: true
I20260812 02:59:44.470726 15102 raft_consensus.cc:2781] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:44.471067 15102 raft_consensus.cc:482] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:44.471031 14671 raft_consensus.cc:2454] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 0.
I20260812 02:59:44.471410 15102 raft_consensus.cc:3037] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:44.476250 15102 raft_consensus.cc:504] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.477598 15102 leader_election.cc:288] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 election: Requested vote from peers 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403), 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547)
I20260812 02:59:44.478343 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702770a8d26b4b269fb3f6cc5ee1d217" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1ab11b7f874fabb66918d268d271fa"
I20260812 02:59:44.478713 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "702770a8d26b4b269fb3f6cc5ee1d217" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9"
I20260812 02:59:44.478931 14952 raft_consensus.cc:3037] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:44.479247 14671 raft_consensus.cc:3037] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:44.485981 14952 raft_consensus.cc:2454] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 1.
I20260812 02:59:44.485980 14671 raft_consensus.cc:2454] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 1.
I20260812 02:59:44.486879 14240 leader_election.cc:302] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:44.487438 15102 raft_consensus.cc:2781] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:44.489223 15102 raft_consensus.cc:686] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 LEADER]: Becoming Leader. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Running, Role: LEADER
I20260812 02:59:44.490052 15102 consensus_queue.cc:237] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } }
I20260812 02:59:44.502040 14076 catalog_manager.cc:5495] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65). New cstate: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } health_report { overall_health: UNKNOWN } } }
W20260812 02:59:44.545306 15051 tablet.cc:2369] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 02:59:44 dist-test-slave-zpvx krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503577, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:44.575086 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.552303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.10.118.126:42029)
0812 02:59:44.552743 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:44.552764 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:44.554315 (+ 1551us) server_negotiation.cc:407] Connection header received
0812 02:59:44.554693 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:44.554715 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:44.554919 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:44.555105 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:44.557118 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.558023 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.559988 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.560706 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.560952 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.565847 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:44.565872 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:44.565893 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:44.565987 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:44.569451 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:44.570501 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:44.570522 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:44.570535 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:44.570730 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:44.571355 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:44.571376 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:44.571389 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:44.571889 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:44.572073 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:44.572874 (+ 801us) server_negotiation.cc:299] Negotiation successful
0812 02:59:44.573553 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":123}
I20260812 02:59:44.575145 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:44.552263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42029 (local address 127.0.0.1:36624)
0812 02:59:44.554138 (+ 1875us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:44.554195 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:44.554474 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:44.555185 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:44.555212 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:44.556213 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:44.556872 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.556905 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.558263 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.558284 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.559735 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:44.559771 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:44.560984 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:44.561023 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:44.561334 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:44.562134 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:44.562211 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:44.565542 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:44.569652 (+ 4110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:44.569679 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:44.569697 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:44.570326 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:44.570913 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:44.570932 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:44.570946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:44.571188 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:44.572055 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:44.572080 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:44.572434 (+ 354us) client_negotiation.cc:769] Sending connection context
0812 02:59:44.572673 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 02:59:44.573166 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1533,"thread_start_us":1160,"threads_started":1}
I20260812 02:59:44.582805 14076 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:36624:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 02:59:44.584805 14076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:44.601497 14354 tablet_service.cc:1467] Processing CreateTablet for tablet d83c93e126724783a67f323f7c12996f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=842c81ca3fe348629707d45e0e02cd5c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:59:44.603258 14354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d83c93e126724783a67f323f7c12996f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:44.602551 14631 tablet_service.cc:1467] Processing CreateTablet for tablet d83c93e126724783a67f323f7c12996f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=842c81ca3fe348629707d45e0e02cd5c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:59:44.603041 14919 tablet_service.cc:1467] Processing CreateTablet for tablet d83c93e126724783a67f323f7c12996f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=842c81ca3fe348629707d45e0e02cd5c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 02:59:44.604140 14631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d83c93e126724783a67f323f7c12996f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:44.604506 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d83c93e126724783a67f323f7c12996f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:44.616088 15082 tablet_bootstrap.cc:492] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: Bootstrap starting.
I20260812 02:59:44.618994 15086 tablet_bootstrap.cc:492] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: Bootstrap starting.
I20260812 02:59:44.619189 15087 tablet_bootstrap.cc:492] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: Bootstrap starting.
I20260812 02:59:44.621233 15082 tablet_bootstrap.cc:654] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:44.625283 15086 tablet_bootstrap.cc:654] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:44.625304 15087 tablet_bootstrap.cc:654] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:44.627898 15082 tablet_bootstrap.cc:492] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: No bootstrap required, opened a new log
I20260812 02:59:44.628502 15082 ts_tablet_manager.cc:1397] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260812 02:59:44.632810 15082 raft_consensus.cc:348] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.633533 15082 raft_consensus.cc:374] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:44.633821 15082 raft_consensus.cc:729] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1ab11b7f874fabb66918d268d271fa, State: Initialized, Role: FOLLOWER
I20260812 02:59:44.634239 15087 tablet_bootstrap.cc:492] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: No bootstrap required, opened a new log
I20260812 02:59:44.634649 15087 ts_tablet_manager.cc:1397] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260812 02:59:44.634495 15082 consensus_queue.cc:260] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.636667 15082 ts_tablet_manager.cc:1428] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260812 02:59:44.637099 15087 raft_consensus.cc:348] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.637813 15087 raft_consensus.cc:374] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:44.636081 15094 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:44.638095 15087 raft_consensus.cc:729] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f2335bee84faa89d4121babfc77d9, State: Initialized, Role: FOLLOWER
I20260812 02:59:44.638417 15094 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:44.639161 15087 consensus_queue.cc:260] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.640470 15086 tablet_bootstrap.cc:492] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: No bootstrap required, opened a new log
I20260812 02:59:44.640899 15086 ts_tablet_manager.cc:1397] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.011s
I20260812 02:59:44.641165 15105 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:44.642043 15105 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:44.643245 15086 raft_consensus.cc:348] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.644029 15086 raft_consensus.cc:374] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:44.644325 15086 raft_consensus.cc:729] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Initialized, Role: FOLLOWER
I20260812 02:59:44.644881 15087 ts_tablet_manager.cc:1428] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260812 02:59:44.645058 15086 consensus_queue.cc:260] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.646278 15102 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:44.646847 15086 ts_tablet_manager.cc:1428] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260812 02:59:44.647035 15102 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:44.923167 15102 raft_consensus.cc:482] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:44.923565 15102 raft_consensus.cc:504] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.924948 15102 leader_election.cc:288] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547), 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403)
I20260812 02:59:44.925891 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83c93e126724783a67f323f7c12996f" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9" is_pre_election: true
I20260812 02:59:44.926113 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83c93e126724783a67f323f7c12996f" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1ab11b7f874fabb66918d268d271fa" is_pre_election: true
I20260812 02:59:44.926476 14671 raft_consensus.cc:2454] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 0.
I20260812 02:59:44.926749 14952 raft_consensus.cc:2454] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 0.
I20260812 02:59:44.927663 14240 leader_election.cc:302] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395f2335bee84faa89d4121babfc77d9, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:44.928300 15102 raft_consensus.cc:2781] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:44.928659 15102 raft_consensus.cc:482] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:44.928969 15102 raft_consensus.cc:3037] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:44.933230 15102 raft_consensus.cc:504] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.934538 15102 leader_election.cc:288] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 election: Requested vote from peers 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547), 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403)
I20260812 02:59:44.935465 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83c93e126724783a67f323f7c12996f" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9"
I20260812 02:59:44.935674 14952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d83c93e126724783a67f323f7c12996f" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1ab11b7f874fabb66918d268d271fa"
I20260812 02:59:44.935911 14671 raft_consensus.cc:3037] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:44.936162 14952 raft_consensus.cc:3037] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:44.940008 14671 raft_consensus.cc:2454] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 1.
I20260812 02:59:44.940237 14952 raft_consensus.cc:2454] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 1.
I20260812 02:59:44.940925 14240 leader_election.cc:302] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395f2335bee84faa89d4121babfc77d9, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:44.941577 15102 raft_consensus.cc:2781] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:44.942020 15102 raft_consensus.cc:686] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [term 1 LEADER]: Becoming Leader. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Running, Role: LEADER
I20260812 02:59:44.942692 15102 consensus_queue.cc:237] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:44.947685 15110 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: TxnStatusTablet state changed. Reason: New leader fb14d2640cf6496ebeea2f59e42cc484. Latest consensus state: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:44.948173 15110 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:59:44.950289 15129 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 02:59:44.950770 15129 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 02:59:44.950793 14076 catalog_manager.cc:5495] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65). New cstate: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } health_report { overall_health: UNKNOWN } } }
I20260812 02:59:45.025998 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.013296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.0.0.1:36894)
0812 02:59:45.013830 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.013863 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.014084 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.017857 (+ 3773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.017888 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.017943 (+ 55us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:59:45.018335 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.018370 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.019782 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.019803 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.021117 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.021155 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.022225 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.022261 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.022586 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.022720 (+ 134us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.023772 (+ 1052us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.023822 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 02:59:45.023980 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 02:59:45.024292 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 02:59:45.026021 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.013504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.10.118.65:35707)
0812 02:59:45.017134 (+ 3630us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.017150 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.017215 (+ 65us) server_negotiation.cc:407] Connection header received
0812 02:59:45.017356 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.017374 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.017598 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.017790 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:45.018633 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.019538 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.021382 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.021959 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.022265 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.023067 (+ 802us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.023574 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.023771 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.024056 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.024238 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3314,"thread_start_us":3047,"threads_started":1}
I20260812 02:59:45.044247 14952 raft_consensus.cc:1261] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Refusing update from remote peer fb14d2640cf6496ebeea2f59e42cc484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:45.045387 14671 raft_consensus.cc:1261] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Refusing update from remote peer fb14d2640cf6496ebeea2f59e42cc484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:45.045835 15143 consensus_queue.cc:1035] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 02:59:45.047295 15110 consensus_queue.cc:1035] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:45.047672 15143 consensus_queue.cc:1035] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:45.088541 15102 consensus_queue.cc:1035] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:45.125993 15105 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: TxnStatusTablet state changed. Reason: New leader fb14d2640cf6496ebeea2f59e42cc484. Latest consensus state: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:45.126765 15105 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:45.137655 15094 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: TxnStatusTablet state changed. Reason: New leader fb14d2640cf6496ebeea2f59e42cc484. Latest consensus state: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:45.139657 15142 mvcc.cc:205] Tried to move back new op lower bound from 7317518684314460160 to 7317518683930988544. Current Snapshot: MvccSnapshot[applied={T|T < 7317518684314460160}]
I20260812 02:59:45.155257 15094 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:45.159521 15102 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:45.160226 15102 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:59:45.169611 15143 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:45.170286 15143 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P fb14d2640cf6496ebeea2f59e42cc484: This TxnStatusTablet replica's current role is: LEADER
I20260812 02:59:45.173019 15160 mvcc.cc:205] Tried to move back new op lower bound from 7317518684314460160 to 7317518683930988544. Current Snapshot: MvccSnapshot[applied={T|T < 7317518684314460160}]
I20260812 02:59:45.179646 15105 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:45.180316 15105 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P 395f2335bee84faa89d4121babfc77d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:45.189018 15094 tablet_replica.cc:439] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } } }
I20260812 02:59:45.189788 15094 tablet_replica.cc:442] T d83c93e126724783a67f323f7c12996f P 9e1ab11b7f874fabb66918d268d271fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 02:59:45.233023 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.208427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.10.118.65:35707)
0812 02:59:45.208919 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.208943 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.217651 (+ 8708us) server_negotiation.cc:407] Connection header received
0812 02:59:45.218260 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.218291 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.218579 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.218707 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:45.220038 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.221391 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.227017 (+ 5626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.228280 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.228863 (+ 583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.230384 (+ 1521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.230713 (+ 329us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.230861 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.231364 (+ 503us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.231547 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 02:59:45.264674 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.252740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:47345 (local address 127.10.118.65:35707)
0812 02:59:45.253229 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.253254 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.255092 (+ 1838us) server_negotiation.cc:407] Connection header received
0812 02:59:45.255565 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.255591 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.255847 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.255996 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:45.256946 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.257791 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.259902 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.260685 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.260955 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.261999 (+ 1044us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.262203 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.262355 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.262613 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.262859 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":109}
I20260812 02:59:45.264966 15166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.252628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.10.118.65:47345)
0812 02:59:45.254874 (+ 2246us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.254944 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.255266 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.256045 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.256072 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.256140 (+ 68us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:59:45.256706 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.256738 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.258008 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.258054 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.259663 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.259699 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.260965 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.261000 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.261512 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.261642 (+ 130us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.262374 (+ 732us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.262425 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 02:59:45.262561 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 02:59:45.262878 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1679,"mutex_wait_us":120,"thread_start_us":1393,"threads_started":1}
I20260812 02:59:45.409950 14076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:45.410658 14076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:45.420675 14076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Sending DeleteTablet for 3 replicas of tablet 702770a8d26b4b269fb3f6cc5ee1d217
I20260812 02:59:45.423022 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet 702770a8d26b4b269fb3f6cc5ee1d217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:45 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:53524
I20260812 02:59:45.423341 14631 tablet_service.cc:1514] Processing DeleteTablet for tablet 702770a8d26b4b269fb3f6cc5ee1d217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:45 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57512
I20260812 02:59:45.423563 14350 tablet_service.cc:1514] Processing DeleteTablet for tablet 702770a8d26b4b269fb3f6cc5ee1d217 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:45 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:36892
I20260812 02:59:45.429386 14076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:59:45.431236 14076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:45.436319 15172 tablet_replica.cc:330] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: stopping tablet replica
I20260812 02:59:45.436786 15170 tablet_replica.cc:330] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: stopping tablet replica
I20260812 02:59:45.437229 15172 raft_consensus.cc:2229] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:45.437847 15172 raft_consensus.cc:2258] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:45.437829 15170 raft_consensus.cc:2229] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:45.438926 15170 raft_consensus.cc:2258] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:45.441457 15172 ts_tablet_manager.cc:1905] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:45.442516 15170 ts_tablet_manager.cc:1905] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:45.439515 15179 tablet_replica.cc:330] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: stopping tablet replica
I20260812 02:59:45.449266 15179 raft_consensus.cc:2229] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:59:45.450294 15179 raft_consensus.cc:2258] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:45.453491 15179 ts_tablet_manager.cc:1905] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:45.459421 14350 tablet_service.cc:1467] Processing CreateTablet for tablet dc7bdda91af94b16b1049b4b15abeb9d (DEFAULT_TABLE table=test-table [id=f6d3c93db54d4d169f62f9ffa4c9f9d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:45.460604 14350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7bdda91af94b16b1049b4b15abeb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:45.461139 14631 tablet_service.cc:1467] Processing CreateTablet for tablet dc7bdda91af94b16b1049b4b15abeb9d (DEFAULT_TABLE table=test-table [id=f6d3c93db54d4d169f62f9ffa4c9f9d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:45.462208 14631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7bdda91af94b16b1049b4b15abeb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:45.469543 14919 tablet_service.cc:1467] Processing CreateTablet for tablet dc7bdda91af94b16b1049b4b15abeb9d (DEFAULT_TABLE table=test-table [id=f6d3c93db54d4d169f62f9ffa4c9f9d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:45.470297 15172 ts_tablet_manager.cc:1918] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:45.470605 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7bdda91af94b16b1049b4b15abeb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:45.470685 15172 log.cc:1196] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/702770a8d26b4b269fb3f6cc5ee1d217
I20260812 02:59:45.471613 15172 ts_tablet_manager.cc:1939] T 702770a8d26b4b269fb3f6cc5ee1d217 P 9e1ab11b7f874fabb66918d268d271fa: Deleting consensus metadata
I20260812 02:59:45.474968 15179 ts_tablet_manager.cc:1918] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:45.474805 14049 catalog_manager.cc:4841] TS 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403): tablet 702770a8d26b4b269fb3f6cc5ee1d217 (table test-table [id=1352140ee24d45ef8c735037d221c672]) successfully deleted
I20260812 02:59:45.475368 15179 log.cc:1196] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/702770a8d26b4b269fb3f6cc5ee1d217
I20260812 02:59:45.479663 15170 ts_tablet_manager.cc:1918] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:45.480085 15170 log.cc:1196] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/702770a8d26b4b269fb3f6cc5ee1d217
I20260812 02:59:45.480880 15170 ts_tablet_manager.cc:1939] T 702770a8d26b4b269fb3f6cc5ee1d217 P 395f2335bee84faa89d4121babfc77d9: Deleting consensus metadata
I20260812 02:59:45.483942 14049 catalog_manager.cc:4841] TS 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547): tablet 702770a8d26b4b269fb3f6cc5ee1d217 (table test-table [id=1352140ee24d45ef8c735037d221c672]) successfully deleted
I20260812 02:59:45.476282 15179 ts_tablet_manager.cc:1939] T 702770a8d26b4b269fb3f6cc5ee1d217 P fb14d2640cf6496ebeea2f59e42cc484: Deleting consensus metadata
I20260812 02:59:45.487699 14045 catalog_manager.cc:4841] TS fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707): tablet 702770a8d26b4b269fb3f6cc5ee1d217 (table test-table [id=1352140ee24d45ef8c735037d221c672]) successfully deleted
I20260812 02:59:45.489043 15180 tablet_bootstrap.cc:492] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: Bootstrap starting.
I20260812 02:59:45.495373 15180 tablet_bootstrap.cc:654] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:45.505505 15182 tablet_bootstrap.cc:492] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: Bootstrap starting.
I20260812 02:59:45.507867 15180 tablet_bootstrap.cc:492] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: No bootstrap required, opened a new log
I20260812 02:59:45.508263 15180 ts_tablet_manager.cc:1397] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260812 02:59:45.510895 15180 raft_consensus.cc:348] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.511653 15180 raft_consensus.cc:374] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:45.511816 15182 tablet_bootstrap.cc:654] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:45.511940 15180 raft_consensus.cc:729] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Initialized, Role: FOLLOWER
I20260812 02:59:45.513301 15180 consensus_queue.cc:260] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.515595 15180 ts_tablet_manager.cc:1428] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20260812 02:59:45.517657 15190 tablet_bootstrap.cc:492] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: Bootstrap starting.
I20260812 02:59:45.523715 15182 tablet_bootstrap.cc:492] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: No bootstrap required, opened a new log
I20260812 02:59:45.524118 15182 ts_tablet_manager.cc:1397] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260812 02:59:45.524509 15190 tablet_bootstrap.cc:654] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:45.526700 15182 raft_consensus.cc:348] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.527462 15182 raft_consensus.cc:374] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:45.527776 15182 raft_consensus.cc:729] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1ab11b7f874fabb66918d268d271fa, State: Initialized, Role: FOLLOWER
I20260812 02:59:45.528759 15182 consensus_queue.cc:260] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.533862 15182 ts_tablet_manager.cc:1428] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20260812 02:59:45.536167 15190 tablet_bootstrap.cc:492] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: No bootstrap required, opened a new log
I20260812 02:59:45.536541 15190 ts_tablet_manager.cc:1397] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20260812 02:59:45.538687 15190 raft_consensus.cc:348] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.539330 15190 raft_consensus.cc:374] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:45.539573 15190 raft_consensus.cc:729] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f2335bee84faa89d4121babfc77d9, State: Initialized, Role: FOLLOWER
I20260812 02:59:45.540153 15190 consensus_queue.cc:260] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.543619 15190 ts_tablet_manager.cc:1428] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:59:45.705909 15094 raft_consensus.cc:482] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:45.706453 15094 raft_consensus.cc:504] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.712810 15094 leader_election.cc:288] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707), 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547)
I20260812 02:59:45.724972 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.709835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:32959 (local address 127.10.118.65:35707)
0812 02:59:45.710523 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.710561 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.712493 (+ 1932us) server_negotiation.cc:407] Connection header received
0812 02:59:45.713212 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.713253 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.713580 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.713758 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:45.715711 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.717122 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.720420 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.722062 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.722451 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.722706 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.723279 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.723555 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260812 02:59:45.725602 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.709704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.10.118.67:32959)
0812 02:59:45.712121 (+ 2417us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.712191 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.712841 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.714469 (+ 1628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.714510 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.714664 (+ 154us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:45.715376 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.715424 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.717466 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.717499 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.720067 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.720124 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.722426 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.722466 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.722885 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.723031 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 02:59:45.723194 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 02:59:45.723464 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"mutex_wait_us":33,"thread_start_us":5228,"threads_started":1}
I20260812 02:59:45.727233 14391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bdda91af94b16b1049b4b15abeb9d" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb14d2640cf6496ebeea2f59e42cc484" is_pre_election: true
I20260812 02:59:45.727617 15105 raft_consensus.cc:482] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:45.728013 14391 raft_consensus.cc:2454] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 0.
I20260812 02:59:45.728083 15105 raft_consensus.cc:504] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.728897 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.711344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:58627 (local address 127.10.118.66:38547)
0812 02:59:45.716287 (+ 4943us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.716317 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.716405 (+ 88us) server_negotiation.cc:407] Connection header received
0812 02:59:45.716563 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.716590 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.716828 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.716981 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:45.718516 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.720199 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.722616 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.724456 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.724841 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.725124 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.726386 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.726667 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4506,"thread_start_us":4679,"threads_started":1}
I20260812 02:59:45.729645 14819 leader_election.cc:302] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:45.730602 15094 raft_consensus.cc:2781] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:45.730932 15094 raft_consensus.cc:482] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:45.731245 15094 raft_consensus.cc:3037] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:45.731403 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.709970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:38547 (local address 127.10.118.67:58627)
0812 02:59:45.715677 (+ 5707us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.715726 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.715961 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.717064 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.717092 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.717199 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:45.718161 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.718222 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.720552 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.720574 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.722267 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.722324 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.725436 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.725473 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.725909 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.726156 (+ 247us) client_negotiation.cc:769] Sending connection context
0812 02:59:45.726311 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 02:59:45.726704 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5255,"mutex_wait_us":79,"thread_start_us":2356,"threads_started":1}
I20260812 02:59:45.733166 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bdda91af94b16b1049b4b15abeb9d" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9" is_pre_election: true
I20260812 02:59:45.734189 14671 raft_consensus.cc:2454] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 0.
I20260812 02:59:45.738204 15094 raft_consensus.cc:504] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.740314 15094 leader_election.cc:288] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 election: Requested vote from peers fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707), 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547)
I20260812 02:59:45.740808 14391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bdda91af94b16b1049b4b15abeb9d" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb14d2640cf6496ebeea2f59e42cc484"
I20260812 02:59:45.741474 14391 raft_consensus.cc:3037] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:45.744648 15105 leader_election.cc:288] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707), 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403)
I20260812 02:59:45.744889 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bdda91af94b16b1049b4b15abeb9d" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9"
I20260812 02:59:45.745465 14671 raft_consensus.cc:3037] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:45.748697 14391 raft_consensus.cc:2454] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 1.
I20260812 02:59:45.749713 14819 leader_election.cc:302] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:45.750481 15094 raft_consensus.cc:2781] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:45.752041 14671 raft_consensus.cc:2454] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 1.
I20260812 02:59:45.759706 15204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.734868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:36403 (local address 127.10.118.66:54155)
0812 02:59:45.744980 (+ 10112us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.745038 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.745305 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.746673 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.746699 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.746807 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:45.747587 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.747641 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.750061 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.750095 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.753408 (+ 3313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.753461 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.756757 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.756787 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.757205 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.757346 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 02:59:45.757499 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 02:59:45.757765 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9662,"mutex_wait_us":35,"thread_start_us":10768,"threads_started":1}
I20260812 02:59:45.762208 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.736890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:54155 (local address 127.10.118.67:36403)
0812 02:59:45.743784 (+ 6894us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.743808 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.745510 (+ 1702us) server_negotiation.cc:407] Connection header received
0812 02:59:45.745663 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.745691 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.745941 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.746096 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:45.747956 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.749723 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.753766 (+ 4043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.755697 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.756146 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.756442 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.760247 (+ 3805us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.760546 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6451,"thread_start_us":6081,"threads_started":1}
I20260812 02:59:45.763397 15094 raft_consensus.cc:686] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 1 LEADER]: Becoming Leader. State: Replica: 9e1ab11b7f874fabb66918d268d271fa, State: Running, Role: LEADER
I20260812 02:59:45.764880 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.732788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:52281 (local address 127.10.118.65:35707)
0812 02:59:45.746123 (+ 13335us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.746157 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.746204 (+ 47us) server_negotiation.cc:407] Connection header received
0812 02:59:45.746345 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.746372 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.746620 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.746779 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 02:59:45.748629 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.750024 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.756176 (+ 6152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.758447 (+ 2271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.758929 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.760511 (+ 1582us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.763121 (+ 2610us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.763442 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12360}
I20260812 02:59:45.764943 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bdda91af94b16b1049b4b15abeb9d" candidate_uuid: "395f2335bee84faa89d4121babfc77d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1ab11b7f874fabb66918d268d271fa" is_pre_election: true
I20260812 02:59:45.764941 15094 consensus_queue.cc:237] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:45.767125 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.731454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.10.118.66:52281)
0812 02:59:45.742188 (+ 10734us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.742247 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.742500 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.747216 (+ 4716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.747250 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.747355 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 02:59:45.748257 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.748309 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.751670 (+ 3361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.751710 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.755798 (+ 4088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.755850 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.759366 (+ 3516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.759411 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.760026 (+ 615us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.760683 (+ 657us) client_negotiation.cc:769] Sending connection context
0812 02:59:45.760862 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 02:59:45.761252 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9858,"thread_start_us":20439,"threads_started":1}
I20260812 02:59:45.769254 14391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7bdda91af94b16b1049b4b15abeb9d" candidate_uuid: "395f2335bee84faa89d4121babfc77d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb14d2640cf6496ebeea2f59e42cc484" is_pre_election: true
I20260812 02:59:45.769896 14391 raft_consensus.cc:2379] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 395f2335bee84faa89d4121babfc77d9 in current term 1: Already voted for candidate 9e1ab11b7f874fabb66918d268d271fa in this term.
I20260812 02:59:45.771142 14539 leader_election.cc:302] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 395f2335bee84faa89d4121babfc77d9; no voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484
I20260812 02:59:45.772444 15105 raft_consensus.cc:2726] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 02:59:45.776860 14078 catalog_manager.cc:5495] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67). New cstate: current_term: 1 leader_uuid: "9e1ab11b7f874fabb66918d268d271fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } health_report { overall_health: HEALTHY } } }
I20260812 02:59:45.860496 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.844206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.10.118.67:36403)
0812 02:59:45.844808 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.844844 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.845565 (+ 721us) server_negotiation.cc:407] Connection header received
0812 02:59:45.846158 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.846195 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.846564 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.846768 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:45.848697 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.849804 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.853032 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.854024 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.854332 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.856171 (+ 1839us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.856956 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.857156 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.857458 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.857642 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 02:59:45.894433 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.880124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:58087 (local address 127.10.118.65:35707)
0812 02:59:45.880592 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.880613 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.882730 (+ 2117us) server_negotiation.cc:407] Connection header received
0812 02:59:45.883273 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.883294 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.883516 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.883683 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:45.885015 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.885888 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.888886 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.889744 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.890234 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.891221 (+ 987us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.891540 (+ 319us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.891690 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:45.892568 (+ 878us) server_negotiation.cc:299] Negotiation successful
0812 02:59:45.892754 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 02:59:45.895119 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.879971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.10.118.67:58087)
0812 02:59:45.882547 (+ 2576us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.882586 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.883014 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.883752 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.883779 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.883831 (+ 52us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:59:45.884747 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.884785 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.886177 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.886213 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.888547 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.888645 (+ 98us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.890145 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.890178 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:45.890635 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:45.890859 (+ 224us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:59:45.892179 (+ 1320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:59:45.892222 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 02:59:45.892479 (+ 257us) client_negotiation.cc:240] Negotiation successful
0812 02:59:45.892812 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2153,"thread_start_us":1890,"threads_started":1}
I20260812 02:59:45.915879 14671 raft_consensus.cc:1261] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Refusing update from remote peer 9e1ab11b7f874fabb66918d268d271fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:45.916462 14391 raft_consensus.cc:1261] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Refusing update from remote peer 9e1ab11b7f874fabb66918d268d271fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:45.917112 15094 consensus_queue.cc:1035] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:45.918363 15206 consensus_queue.cc:1035] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 02:59:46.005650 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.994530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:54887 (local address 127.10.118.67:36403)
0812 02:59:45.995050 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:45.995085 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:45.996267 (+ 1182us) server_negotiation.cc:407] Connection header received
0812 02:59:45.996747 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:45.996776 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:45.997073 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:45.997255 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:45.998361 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.999351 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:46.001299 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:46.001887 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:46.002174 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:46.002923 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:46.003122 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:46.003280 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:46.003894 (+ 614us) server_negotiation.cc:299] Negotiation successful
0812 02:59:46.004083 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260812 02:59:46.005722 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:45.994303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:36403 (local address 127.10.118.65:54887)
0812 02:59:45.996033 (+ 1730us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:45.996126 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:45.996465 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:45.997360 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:45.997389 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:45.997443 (+ 54us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 02:59:45.998113 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:45.998149 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:45.999558 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:45.999597 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:46.001051 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:46.001087 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:46.002159 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:46.002192 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:46.002506 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:46.002624 (+ 118us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 02:59:46.003313 (+ 689us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 02:59:46.003357 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 02:59:46.003493 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 02:59:46.003805 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":615,"threads_started":1}
W20260812 02:59:46.008409 15146 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:59:47.840180 14072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:47.841106 14072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:47.856297 14072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Sending DeleteTablet for 3 replicas of tablet dc7bdda91af94b16b1049b4b15abeb9d
I20260812 02:59:47.858227 14350 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7bdda91af94b16b1049b4b15abeb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:47 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:36892
I20260812 02:59:47.860185 14631 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7bdda91af94b16b1049b4b15abeb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:47 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57512
I20260812 02:59:47.861356 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7bdda91af94b16b1049b4b15abeb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:47 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:53524
I20260812 02:59:47.869334 15284 tablet_replica.cc:330] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: stopping tablet replica
I20260812 02:59:47.870175 15284 raft_consensus.cc:2229] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:47.870760 15284 raft_consensus.cc:2258] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:47.873826 15284 ts_tablet_manager.cc:1905] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:47.888654 15284 ts_tablet_manager.cc:1918] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:59:47.889142 15284 log.cc:1196] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/dc7bdda91af94b16b1049b4b15abeb9d
I20260812 02:59:47.890015 15284 ts_tablet_manager.cc:1939] T dc7bdda91af94b16b1049b4b15abeb9d P fb14d2640cf6496ebeea2f59e42cc484: Deleting consensus metadata
I20260812 02:59:47.892927 14045 catalog_manager.cc:4841] TS fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707): tablet dc7bdda91af94b16b1049b4b15abeb9d (table test-table [id=f6d3c93db54d4d169f62f9ffa4c9f9d5]) successfully deleted
I20260812 02:59:47.894871 15287 tablet_replica.cc:330] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: stopping tablet replica
I20260812 02:59:47.896013 15286 tablet_replica.cc:330] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: stopping tablet replica
I20260812 02:59:47.897348 15286 raft_consensus.cc:2229] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:47.897879 15287 raft_consensus.cc:2229] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:59:47.898979 15287 raft_consensus.cc:2258] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:47.899531 15286 raft_consensus.cc:2258] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:47.902333 15286 ts_tablet_manager.cc:1905] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:47.904361 15287 ts_tablet_manager.cc:1905] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:47.931206 15286 ts_tablet_manager.cc:1918] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:59:47.934099 15287 ts_tablet_manager.cc:1918] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:59:47.934562 15286 log.cc:1196] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/dc7bdda91af94b16b1049b4b15abeb9d
Aug 12 02:59:47 dist-test-slave-zpvx krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260812 02:59:47.935873 15286 ts_tablet_manager.cc:1939] T dc7bdda91af94b16b1049b4b15abeb9d P 395f2335bee84faa89d4121babfc77d9: Deleting consensus metadata
I20260812 02:59:47.935205 15287 log.cc:1196] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/dc7bdda91af94b16b1049b4b15abeb9d
I20260812 02:59:47.936962 15287 ts_tablet_manager.cc:1939] T dc7bdda91af94b16b1049b4b15abeb9d P 9e1ab11b7f874fabb66918d268d271fa: Deleting consensus metadata
I20260812 02:59:47.939963 14049 catalog_manager.cc:4841] TS 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547): tablet dc7bdda91af94b16b1049b4b15abeb9d (table test-table [id=f6d3c93db54d4d169f62f9ffa4c9f9d5]) successfully deleted
I20260812 02:59:47.943084 14049 catalog_manager.cc:4841] TS 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403): tablet dc7bdda91af94b16b1049b4b15abeb9d (table test-table [id=f6d3c93db54d4d169f62f9ffa4c9f9d5]) successfully deleted
Aug 12 02:59:47 dist-test-slave-zpvx krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:48.004665 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:47.951453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.10.118.126:42029)
0812 02:59:47.964440 (+ 12987us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:47.964480 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:47.966040 (+ 1560us) server_negotiation.cc:407] Connection header received
0812 02:59:47.966523 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:47.966554 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:47.966825 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:47.967049 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:47.977236 (+ 10187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:47.979090 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:47.981980 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:47.982967 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:47.983262 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:47.991061 (+ 7799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:47.991115 (+ 54us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:47.991140 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:47.991272 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:47.997224 (+ 5952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:47.998681 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:47.998715 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:47.998737 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:47.998994 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:47.999785 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:47.999815 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:47.999836 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:48.000571 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:48.001002 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:48.001770 (+ 768us) server_negotiation.cc:299] Negotiation successful
0812 02:59:48.002253 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12377,"thread_start_us":12931,"threads_started":1}
I20260812 02:59:48.012207 14078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 02:59:48.014456 14078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:48.043740 14631 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 (DEFAULT_TABLE table=test-table [id=4495d870d7e24c3d93129740e972d1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:48.044911 14631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:48.049984 14350 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 (DEFAULT_TABLE table=test-table [id=4495d870d7e24c3d93129740e972d1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:48.051057 14350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:48.060123 14919 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 (DEFAULT_TABLE table=test-table [id=4495d870d7e24c3d93129740e972d1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:48.061329 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:48.079171 15303 tablet_bootstrap.cc:492] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: Bootstrap starting.
I20260812 02:59:48.092316 15303 tablet_bootstrap.cc:654] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:48.113864 15303 tablet_bootstrap.cc:492] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: No bootstrap required, opened a new log
I20260812 02:59:48.114408 15303 ts_tablet_manager.cc:1397] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.006s
I20260812 02:59:48.117070 15303 raft_consensus.cc:348] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.118017 15303 raft_consensus.cc:374] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:48.118346 15303 raft_consensus.cc:729] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Initialized, Role: FOLLOWER
I20260812 02:59:48.119232 15303 consensus_queue.cc:260] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.130643 15310 tablet_bootstrap.cc:492] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: Bootstrap starting.
I20260812 02:59:48.133936 15303 ts_tablet_manager.cc:1428] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20260812 02:59:48.137763 15307 tablet_bootstrap.cc:492] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: Bootstrap starting.
I20260812 02:59:48.161957 15310 tablet_bootstrap.cc:654] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:48.163779 15307 tablet_bootstrap.cc:654] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:48.185735 15307 tablet_bootstrap.cc:492] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: No bootstrap required, opened a new log
I20260812 02:59:48.186158 15307 ts_tablet_manager.cc:1397] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.010s
I20260812 02:59:48.196625 15307 raft_consensus.cc:348] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.197537 15307 raft_consensus.cc:374] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:48.197919 15307 raft_consensus.cc:729] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1ab11b7f874fabb66918d268d271fa, State: Initialized, Role: FOLLOWER
I20260812 02:59:48.198558 15310 tablet_bootstrap.cc:492] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: No bootstrap required, opened a new log
I20260812 02:59:48.198964 15310 ts_tablet_manager.cc:1397] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: Time spent bootstrapping tablet: real 0.069s user 0.014s sys 0.000s
I20260812 02:59:48.198621 15307 consensus_queue.cc:260] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.201031 15307 ts_tablet_manager.cc:1428] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20260812 02:59:48.212507 15310 raft_consensus.cc:348] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.221268 15310 raft_consensus.cc:374] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:48.221657 15310 raft_consensus.cc:729] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f2335bee84faa89d4121babfc77d9, State: Initialized, Role: FOLLOWER
I20260812 02:59:48.222510 15310 consensus_queue.cc:260] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.240888 15310 ts_tablet_manager.cc:1428] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: Time spent starting tablet: real 0.042s user 0.009s sys 0.000s
I20260812 02:59:48.478147 15227 raft_consensus.cc:482] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:48.478693 15227 raft_consensus.cc:504] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.480643 15227 leader_election.cc:288] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707), 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547)
I20260812 02:59:48.481518 14391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd5b44e0d247fb8e9dd7d4efb40291" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb14d2640cf6496ebeea2f59e42cc484" is_pre_election: true
I20260812 02:59:48.482128 14391 raft_consensus.cc:2454] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 0.
I20260812 02:59:48.482187 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd5b44e0d247fb8e9dd7d4efb40291" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9" is_pre_election: true
I20260812 02:59:48.483059 14671 raft_consensus.cc:2454] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 0.
I20260812 02:59:48.483479 14819 leader_election.cc:302] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:48.484344 15227 raft_consensus.cc:2781] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:48.484694 15227 raft_consensus.cc:482] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:48.485001 15227 raft_consensus.cc:3037] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:48.491957 15227 raft_consensus.cc:504] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.493651 15227 leader_election.cc:288] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 election: Requested vote from peers fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707), 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547)
I20260812 02:59:48.494477 14391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd5b44e0d247fb8e9dd7d4efb40291" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb14d2640cf6496ebeea2f59e42cc484"
I20260812 02:59:48.495020 14391 raft_consensus.cc:3037] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:48.495069 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdd5b44e0d247fb8e9dd7d4efb40291" candidate_uuid: "9e1ab11b7f874fabb66918d268d271fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9"
I20260812 02:59:48.495560 14671 raft_consensus.cc:3037] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:48.501713 14391 raft_consensus.cc:2454] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 1.
I20260812 02:59:48.501868 14671 raft_consensus.cc:2454] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e1ab11b7f874fabb66918d268d271fa in term 1.
I20260812 02:59:48.502926 14819 leader_election.cc:302] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:48.504066 15227 raft_consensus.cc:2781] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:48.505434 15227 raft_consensus.cc:686] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 1 LEADER]: Becoming Leader. State: Replica: 9e1ab11b7f874fabb66918d268d271fa, State: Running, Role: LEADER
I20260812 02:59:48.506453 15227 consensus_queue.cc:237] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.516091 14078 catalog_manager.cc:5495] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67). New cstate: current_term: 1 leader_uuid: "9e1ab11b7f874fabb66918d268d271fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } health_report { overall_health: HEALTHY } } }
I20260812 02:59:48.591457 15342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:48.578454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.10.118.67:36403)
0812 02:59:48.581402 (+ 2948us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:48.581429 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:48.582003 (+ 574us) server_negotiation.cc:407] Connection header received
0812 02:59:48.582409 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:48.582428 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:48.582789 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:48.582955 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:48.583815 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:48.584819 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:48.587400 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:48.588064 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:48.588346 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:48.589248 (+ 902us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:48.589450 (+ 202us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:48.589607 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:48.590019 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 02:59:48.590213 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2522,"thread_start_us":2247,"threads_started":1}
I20260812 02:59:48.616503 14671 raft_consensus.cc:1261] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Refusing update from remote peer 9e1ab11b7f874fabb66918d268d271fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:48.618090 15227 consensus_queue.cc:1035] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:48.650684 14391 raft_consensus.cc:1261] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Refusing update from remote peer 9e1ab11b7f874fabb66918d268d271fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:48.653174 15227 consensus_queue.cc:1035] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 02:59:48.771819 14072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:48.772331 14072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:48.779702 14072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Sending DeleteTablet for 3 replicas of tablet 3fdd5b44e0d247fb8e9dd7d4efb40291
I20260812 02:59:48.781414 14350 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:48 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:36892
I20260812 02:59:48.782189 14631 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:48 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57512
I20260812 02:59:48.784324 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:48 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:53524
I20260812 02:59:48.788337 14072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 02:59:48.791312 15365 tablet_replica.cc:330] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: stopping tablet replica
W20260812 02:59:48.791309 14072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 02:59:48.791730 15364 tablet_replica.cc:330] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: stopping tablet replica
I20260812 02:59:48.792619 15364 raft_consensus.cc:2229] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:48.793190 15364 raft_consensus.cc:2258] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:48.793599 15365 raft_consensus.cc:2229] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 1 LEADER]: Raft consensus shutting down.
I20260812 02:59:48.794876 15365 raft_consensus.cc:2258] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:48.795675 15362 tablet_replica.cc:330] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: stopping tablet replica
I20260812 02:59:48.796562 15362 raft_consensus.cc:2229] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:48.797197 15362 raft_consensus.cc:2258] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:48.799028 15365 ts_tablet_manager.cc:1905] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:48.800037 15362 ts_tablet_manager.cc:1905] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:48.814529 15365 ts_tablet_manager.cc:1918] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:48.814971 15365 log.cc:1196] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/3fdd5b44e0d247fb8e9dd7d4efb40291
I20260812 02:59:48.815820 15365 ts_tablet_manager.cc:1939] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 9e1ab11b7f874fabb66918d268d271fa: Deleting consensus metadata
I20260812 02:59:48.817216 15362 ts_tablet_manager.cc:1918] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:48.817657 15362 log.cc:1196] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/3fdd5b44e0d247fb8e9dd7d4efb40291
I20260812 02:59:48.818575 15362 ts_tablet_manager.cc:1939] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P fb14d2640cf6496ebeea2f59e42cc484: Deleting consensus metadata
I20260812 02:59:48.818594 14049 catalog_manager.cc:4841] TS 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403): tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 (table test-table [id=4495d870d7e24c3d93129740e972d1ba]) successfully deleted
I20260812 02:59:48.821653 14045 catalog_manager.cc:4841] TS fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707): tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 (table test-table [id=4495d870d7e24c3d93129740e972d1ba]) successfully deleted
I20260812 02:59:48.822037 14350 tablet_service.cc:1467] Processing CreateTablet for tablet 928bc41abc5a4b13a2c5e66fe98db229 (DEFAULT_TABLE table=test-table [id=28d0f8332fa346fd9b383ba86e892be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:48.823505 14350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928bc41abc5a4b13a2c5e66fe98db229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:48.824103 14631 tablet_service.cc:1467] Processing CreateTablet for tablet 928bc41abc5a4b13a2c5e66fe98db229 (DEFAULT_TABLE table=test-table [id=28d0f8332fa346fd9b383ba86e892be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:48.824448 15364 ts_tablet_manager.cc:1905] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:48.825227 14631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928bc41abc5a4b13a2c5e66fe98db229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:48.829305 14919 tablet_service.cc:1467] Processing CreateTablet for tablet 928bc41abc5a4b13a2c5e66fe98db229 (DEFAULT_TABLE table=test-table [id=28d0f8332fa346fd9b383ba86e892be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 02:59:48.830386 14919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928bc41abc5a4b13a2c5e66fe98db229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:48.841854 15364 ts_tablet_manager.cc:1918] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 02:59:48.842396 15364 log.cc:1196] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/3fdd5b44e0d247fb8e9dd7d4efb40291
I20260812 02:59:48.843201 15364 ts_tablet_manager.cc:1939] T 3fdd5b44e0d247fb8e9dd7d4efb40291 P 395f2335bee84faa89d4121babfc77d9: Deleting consensus metadata
I20260812 02:59:48.846294 14049 catalog_manager.cc:4841] TS 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547): tablet 3fdd5b44e0d247fb8e9dd7d4efb40291 (table test-table [id=4495d870d7e24c3d93129740e972d1ba]) successfully deleted
I20260812 02:59:48.849606 15368 tablet_bootstrap.cc:492] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484: Bootstrap starting.
I20260812 02:59:48.855132 15368 tablet_bootstrap.cc:654] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:48.871449 15369 tablet_bootstrap.cc:492] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: Bootstrap starting.
I20260812 02:59:48.874485 15373 tablet_bootstrap.cc:492] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: Bootstrap starting.
I20260812 02:59:48.875386 15368 tablet_bootstrap.cc:492] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484: No bootstrap required, opened a new log
I20260812 02:59:48.875897 15368 ts_tablet_manager.cc:1397] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.020s
I20260812 02:59:48.878909 15368 raft_consensus.cc:348] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.880005 15368 raft_consensus.cc:374] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:48.880434 15368 raft_consensus.cc:729] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Initialized, Role: FOLLOWER
I20260812 02:59:48.880335 15373 tablet_bootstrap.cc:654] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:48.881762 15369 tablet_bootstrap.cc:654] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:48.881225 15368 consensus_queue.cc:260] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.889882 15368 ts_tablet_manager.cc:1428] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260812 02:59:48.896441 15369 tablet_bootstrap.cc:492] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: No bootstrap required, opened a new log
I20260812 02:59:48.897022 15369 ts_tablet_manager.cc:1397] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.000s
I20260812 02:59:48.899237 15373 tablet_bootstrap.cc:492] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: No bootstrap required, opened a new log
I20260812 02:59:48.899729 15373 ts_tablet_manager.cc:1397] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.016s
I20260812 02:59:48.899032 15369 raft_consensus.cc:348] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.900203 15369 raft_consensus.cc:374] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:48.902161 15369 raft_consensus.cc:729] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395f2335bee84faa89d4121babfc77d9, State: Initialized, Role: FOLLOWER
I20260812 02:59:48.902292 15373 raft_consensus.cc:348] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.903012 15373 raft_consensus.cc:374] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:48.903316 15373 raft_consensus.cc:729] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1ab11b7f874fabb66918d268d271fa, State: Initialized, Role: FOLLOWER
I20260812 02:59:48.902957 15369 consensus_queue.cc:260] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.904038 15373 consensus_queue.cc:260] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:48.905114 15369 ts_tablet_manager.cc:1428] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260812 02:59:48.907014 15373 ts_tablet_manager.cc:1428] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260812 02:59:49.052044 15348 raft_consensus.cc:482] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 02:59:49.052569 15348 raft_consensus.cc:504] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:49.054500 15348 leader_election.cc:288] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547), 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403)
I20260812 02:59:49.055678 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928bc41abc5a4b13a2c5e66fe98db229" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9" is_pre_election: true
I20260812 02:59:49.055711 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928bc41abc5a4b13a2c5e66fe98db229" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1ab11b7f874fabb66918d268d271fa" is_pre_election: true
I20260812 02:59:49.056255 14671 raft_consensus.cc:2454] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 0.
I20260812 02:59:49.056362 14950 raft_consensus.cc:2454] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 0.
I20260812 02:59:49.057379 14240 leader_election.cc:302] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 395f2335bee84faa89d4121babfc77d9, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:49.058120 15348 raft_consensus.cc:2781] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 02:59:49.058473 15348 raft_consensus.cc:482] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 02:59:49.058766 15348 raft_consensus.cc:3037] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:49.065752 15348 raft_consensus.cc:504] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:49.067548 15348 leader_election.cc:288] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 election: Requested vote from peers 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547), 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403)
I20260812 02:59:49.068691 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928bc41abc5a4b13a2c5e66fe98db229" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e1ab11b7f874fabb66918d268d271fa"
I20260812 02:59:49.069242 14950 raft_consensus.cc:3037] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:49.069811 14671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928bc41abc5a4b13a2c5e66fe98db229" candidate_uuid: "fb14d2640cf6496ebeea2f59e42cc484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "395f2335bee84faa89d4121babfc77d9"
I20260812 02:59:49.070329 14671 raft_consensus.cc:3037] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:49.076022 14950 raft_consensus.cc:2454] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 1.
I20260812 02:59:49.077013 14240 leader_election.cc:302] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e1ab11b7f874fabb66918d268d271fa, fb14d2640cf6496ebeea2f59e42cc484; no voters:
I20260812 02:59:49.076107 14671 raft_consensus.cc:2454] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb14d2640cf6496ebeea2f59e42cc484 in term 1.
I20260812 02:59:49.078017 15348 raft_consensus.cc:2781] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:49.079288 15348 raft_consensus.cc:686] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [term 1 LEADER]: Becoming Leader. State: Replica: fb14d2640cf6496ebeea2f59e42cc484, State: Running, Role: LEADER
I20260812 02:59:49.080076 15348 consensus_queue.cc:237] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } }
I20260812 02:59:49.088418 14072 catalog_manager.cc:5495] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65). New cstate: current_term: 1 leader_uuid: "fb14d2640cf6496ebeea2f59e42cc484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb14d2640cf6496ebeea2f59e42cc484" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 35707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 } health_report { overall_health: UNKNOWN } } }
I20260812 02:59:49.228874 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:49.202591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.10.118.65:35707)
0812 02:59:49.212727 (+ 10136us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:49.212765 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:49.212850 (+ 85us) server_negotiation.cc:407] Connection header received
0812 02:59:49.213155 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:49.213183 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:49.213518 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:49.213706 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 02:59:49.216607 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:49.218482 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:49.221398 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:49.223353 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:49.223775 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:49.225106 (+ 1331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 02:59:49.225386 (+ 280us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 02:59:49.225544 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:49.226266 (+ 722us) server_negotiation.cc:299] Negotiation successful
0812 02:59:49.226500 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9481,"thread_start_us":9131,"threads_started":1}
I20260812 02:59:49.259516 14950 raft_consensus.cc:1261] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Refusing update from remote peer fb14d2640cf6496ebeea2f59e42cc484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:49.259863 14671 raft_consensus.cc:1261] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Refusing update from remote peer fb14d2640cf6496ebeea2f59e42cc484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 02:59:49.261212 15327 consensus_queue.cc:1035] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "395f2335bee84faa89d4121babfc77d9" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 02:59:49.261988 15348 consensus_queue.cc:1035] T 928bc41abc5a4b13a2c5e66fe98db229 P fb14d2640cf6496ebeea2f59e42cc484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e1ab11b7f874fabb66918d268d271fa" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 02:59:49.347800 15353 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 02:59:51.064718 14072 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:51.065349 14072 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 02:59:51.076733 14072 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf726659b71642368fff36ad5fbafeba: Sending DeleteTablet for 3 replicas of tablet 928bc41abc5a4b13a2c5e66fe98db229
I20260812 02:59:51.078519 14354 tablet_service.cc:1514] Processing DeleteTablet for tablet 928bc41abc5a4b13a2c5e66fe98db229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:51 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:36892
I20260812 02:59:51.079468 14919 tablet_service.cc:1514] Processing DeleteTablet for tablet 928bc41abc5a4b13a2c5e66fe98db229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:51 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:53524
I20260812 02:59:51.079751 14631 tablet_service.cc:1514] Processing DeleteTablet for tablet 928bc41abc5a4b13a2c5e66fe98db229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 02:59:51 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57512
I20260812 02:59:51.083873 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 14205
I20260812 02:59:51.097312 15439 tablet_replica.cc:330] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: stopping tablet replica
I20260812 02:59:51.098026 15439 raft_consensus.cc:2229] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:51.098512 15439 raft_consensus.cc:2258] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:51.102660 15439 ts_tablet_manager.cc:1905] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:51.118124 15439 ts_tablet_manager.cc:1918] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:59:51.118603 15439 log.cc:1196] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/928bc41abc5a4b13a2c5e66fe98db229
I20260812 02:59:51.119403 15439 ts_tablet_manager.cc:1939] T 928bc41abc5a4b13a2c5e66fe98db229 P 9e1ab11b7f874fabb66918d268d271fa: Deleting consensus metadata
I20260812 02:59:51.122684 14049 catalog_manager.cc:4841] TS 9e1ab11b7f874fabb66918d268d271fa (127.10.118.67:36403): tablet 928bc41abc5a4b13a2c5e66fe98db229 (table test-table [id=28d0f8332fa346fd9b383ba86e892be5]) successfully deleted
I20260812 02:59:51.125696 15440 tablet_replica.cc:330] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: stopping tablet replica
I20260812 02:59:51.126797 15440 raft_consensus.cc:2229] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 02:59:51.127652 15440 raft_consensus.cc:2258] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 02:59:51.131759 15440 ts_tablet_manager.cc:1905] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 02:59:51.147709 15440 ts_tablet_manager.cc:1918] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 02:59:51.148260 15440 log.cc:1196] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/928bc41abc5a4b13a2c5e66fe98db229
I20260812 02:59:51.149260 15440 ts_tablet_manager.cc:1939] T 928bc41abc5a4b13a2c5e66fe98db229 P 395f2335bee84faa89d4121babfc77d9: Deleting consensus metadata
I20260812 02:59:51.152341 14049 catalog_manager.cc:4841] TS 395f2335bee84faa89d4121babfc77d9 (127.10.118.66:38547): tablet 928bc41abc5a4b13a2c5e66fe98db229 (table test-table [id=28d0f8332fa346fd9b383ba86e892be5]) successfully deleted
W20260812 02:59:51.165072 14045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:35707 (error 108)
I20260812 02:59:51.167158 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 14504
I20260812 02:59:51.185513 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:51.180927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.0.0.1:47600)
0812 02:59:51.184480 (+ 3553us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:51.184728 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3044,"thread_start_us":2255,"threads_started":1}
W20260812 02:59:51.185951 14045 catalog_manager.cc:4584] TS fb14d2640cf6496ebeea2f59e42cc484 (127.10.118.65:35707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 928bc41abc5a4b13a2c5e66fe98db229: Network error: Client connection negotiation failed: client connection to 127.10.118.65:35707: connect: Connection refused (error 111)
I20260812 02:59:51.218708 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 14787
I20260812 02:59:51.241747 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:51.240022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.0.0.1:47614)
0812 02:59:51.240700 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:51.240945 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260812 02:59:51.244680 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:51.243200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:35707 (local address 127.0.0.1:47630)
0812 02:59:51.243819 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:51.244047 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:35707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 02:59:51.271452 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 14001
2026-08-12T02:59:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (16568 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 02:59:51 dist-test-slave-zpvx krb5kdc[15451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 02:59:51 dist-test-slave-zpvx krb5kdc[15451](info): set up 2 sockets
Aug 12 02:59:51 dist-test-slave-zpvx krb5kdc[15451](info): commencing operation
krb5kdc: starting...
W20260812 02:59:52.678409 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 02:59:52 dist-test-slave-zpvx krb5kdc[15451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T02:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T02:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 02:59:52.893090 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:33183
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:33183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:53.195182 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:53.195807 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:53.196451 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:53.226190 15478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 02:59:53.226578 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:53.226890 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:53.227151 15478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 02:59:53.227406 15478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 02:59:53.263219 15478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:33183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:33183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:53.264523 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:53.265595 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:53.284590 15490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:53.284629 15491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:53.284711 15495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:53.286116 15478 server_base.cc:1034] running on GCE node
I20260812 02:59:54.473645 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:54.476974 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:54.478415 15478 hybrid_clock.cc:648] HybridClock initialized: now 1786503594478368 us; error 66 us; skew 500 ppm
I20260812 02:59:54.479278 15478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:54 dist-test-slave-zpvx krb5kdc[15451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503594, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:54.485620 15478 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 02:59:54.491544 15478 webserver.cc:460] Webserver started at http://127.10.118.126:41133/ using document root <none> and password file <none>
I20260812 02:59:54.492575 15478 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:54.492841 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:54.493330 15478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:54.498775 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "b407c9f104bb4342af7d520e8bfa8214"
format_stamp: "Formatted at 2026-08-12 02:59:54 on dist-test-slave-zpvx"
server_key: "f8675b91e77efb28f0f3be14b92abdbe"
server_key_iv: "59a0f9ff15ce47e8e004ad9d08bbd9eb"
server_key_version: "encryptionkey@0"
I20260812 02:59:54.499949 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "b407c9f104bb4342af7d520e8bfa8214"
format_stamp: "Formatted at 2026-08-12 02:59:54 on dist-test-slave-zpvx"
server_key: "f8675b91e77efb28f0f3be14b92abdbe"
server_key_iv: "59a0f9ff15ce47e8e004ad9d08bbd9eb"
server_key_version: "encryptionkey@0"
I20260812 02:59:54.507337 15478 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 02:59:54.512919 15511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:54.513931 15478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 02:59:54.514279 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "b407c9f104bb4342af7d520e8bfa8214"
format_stamp: "Formatted at 2026-08-12 02:59:54 on dist-test-slave-zpvx"
server_key: "f8675b91e77efb28f0f3be14b92abdbe"
server_key_iv: "59a0f9ff15ce47e8e004ad9d08bbd9eb"
server_key_version: "encryptionkey@0"
I20260812 02:59:54.514673 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:54.551030 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:54.556730 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:54.557219 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:54.627609 15478 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:33183
I20260812 02:59:54.627702 15624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:33183 every 8 connection(s)
I20260812 02:59:54.630422 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 02:59:54.635963 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 15478
I20260812 02:59:54.636302 15626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 02:59:54.636557 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 02:59:54.638229 10713 external_mini_cluster.cc:1428] Setting key d24d71bbcd54d102dad9943e93009794
I20260812 02:59:54.664021 15626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214: Bootstrap starting.
Aug 12 02:59:54 dist-test-slave-zpvx krb5kdc[15451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 02:59:54.670069 15626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214: Neither blocks nor log segments found. Creating new log.
I20260812 02:59:54.672175 15626 log.cc:824] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214: Log is configured to *not* fsync() on all Append() calls
I20260812 02:59:54.677731 15626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214: No bootstrap required, opened a new log
I20260812 02:59:54.677918 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:54.641317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.10.118.126:33183)
0812 02:59:54.644099 (+ 2782us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:54.644130 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:54.644183 (+ 53us) server_negotiation.cc:407] Connection header received
0812 02:59:54.649090 (+ 4907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:54.649132 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:54.649471 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:54.649803 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:54.653267 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:54.654885 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:54.657711 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:54.658571 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:54.658820 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:54.666135 (+ 7315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:54.666159 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:54.666190 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:54.666295 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:54.670532 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:54.671976 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:54.672001 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:54.672017 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:54.672251 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:54.673021 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:54.673040 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:54.673054 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:54.674044 (+ 990us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:54.674561 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:54.675286 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 02:59:54.675962 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2340,"thread_start_us":1832,"threads_started":1}
I20260812 02:59:54.701483 15626 raft_consensus.cc:348] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b407c9f104bb4342af7d520e8bfa8214" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33183 } }
I20260812 02:59:54.702307 15626 raft_consensus.cc:374] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 02:59:54.702611 15626 raft_consensus.cc:729] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b407c9f104bb4342af7d520e8bfa8214, State: Initialized, Role: FOLLOWER
I20260812 02:59:54.703421 15626 consensus_queue.cc:260] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b407c9f104bb4342af7d520e8bfa8214" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33183 } }
I20260812 02:59:54.703948 15626 raft_consensus.cc:388] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 02:59:54.704252 15626 raft_consensus.cc:482] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 02:59:54.704632 15626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 0 FOLLOWER]: Advancing to term 1
I20260812 02:59:54.708640 15626 raft_consensus.cc:504] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b407c9f104bb4342af7d520e8bfa8214" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33183 } }
I20260812 02:59:54.709355 15626 leader_election.cc:302] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b407c9f104bb4342af7d520e8bfa8214; no voters:
I20260812 02:59:54.711155 15626 leader_election.cc:288] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 02:59:54.711627 15636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 02:59:54.715539 15626 sys_catalog.cc:564] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 02:59:54.716243 15636 raft_consensus.cc:686] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [term 1 LEADER]: Becoming Leader. State: Replica: b407c9f104bb4342af7d520e8bfa8214, State: Running, Role: LEADER
I20260812 02:59:54.716964 15636 consensus_queue.cc:237] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b407c9f104bb4342af7d520e8bfa8214" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33183 } }
I20260812 02:59:54.728909 15637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b407c9f104bb4342af7d520e8bfa8214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b407c9f104bb4342af7d520e8bfa8214" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33183 } } }
I20260812 02:59:54.730072 15640 sys_catalog.cc:455] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b407c9f104bb4342af7d520e8bfa8214. Latest consensus state: current_term: 1 leader_uuid: "b407c9f104bb4342af7d520e8bfa8214" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b407c9f104bb4342af7d520e8bfa8214" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33183 } } }
I20260812 02:59:54.730701 15640 sys_catalog.cc:458] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:54.731896 15637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214 [sys.catalog]: This master's current role is: LEADER
I20260812 02:59:54.740465 15651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 02:59:54.749584 15651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 02:59:54.764763 15651 catalog_manager.cc:1347] Generated new cluster ID: c761621e7d7442c8bb50ae1ff32bb656
I20260812 02:59:54.765044 15651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 02:59:54.805881 15651 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 02:59:54.808516 15651 catalog_manager.cc:1504] Loading token signing keys...
I20260812 02:59:54.826495 15651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b407c9f104bb4342af7d520e8bfa8214: Generated new TSK 0
I20260812 02:59:54.827589 15651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 02:59:54.897243 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:33183
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:55.208742 15671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:55.209627 15671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:55.210315 15671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:55.242619 15671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:55.243027 15671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:55.243970 15671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 02:59:55.280109 15671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:55.281356 15671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:55.282404 15671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:55.303225 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 02:59:55.304369 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:55.305157 15671 server_base.cc:1034] running on GCE node
W20260812 02:59:55.304925 15684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:56.508626 15671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:56.510937 15671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:56.512293 15671 hybrid_clock.cc:648] HybridClock initialized: now 1786503596512258 us; error 46 us; skew 500 ppm
I20260812 02:59:56.513188 15671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:56 dist-test-slave-zpvx krb5kdc[15451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:56.524188 15671 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 02:59:56.530645 15671 webserver.cc:460] Webserver started at http://127.10.118.65:46873/ using document root <none> and password file <none>
I20260812 02:59:56.531659 15671 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:56.531930 15671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:56.532511 15671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:56.538341 15671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "67c8c2c1636345a4b47674e5f0a59fd5"
format_stamp: "Formatted at 2026-08-12 02:59:56 on dist-test-slave-zpvx"
server_key: "0ef5bbdd5cd47e22a1dd72960eed0199"
server_key_iv: "99c0150af75ff61549cbe15b0b0a63be"
server_key_version: "encryptionkey@0"
I20260812 02:59:56.539649 15671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "67c8c2c1636345a4b47674e5f0a59fd5"
format_stamp: "Formatted at 2026-08-12 02:59:56 on dist-test-slave-zpvx"
server_key: "0ef5bbdd5cd47e22a1dd72960eed0199"
server_key_iv: "99c0150af75ff61549cbe15b0b0a63be"
server_key_version: "encryptionkey@0"
I20260812 02:59:56.547420 15671 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 02:59:56.553408 15697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:56.554594 15671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 02:59:56.554957 15671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "67c8c2c1636345a4b47674e5f0a59fd5"
format_stamp: "Formatted at 2026-08-12 02:59:56 on dist-test-slave-zpvx"
server_key: "0ef5bbdd5cd47e22a1dd72960eed0199"
server_key_iv: "99c0150af75ff61549cbe15b0b0a63be"
server_key_version: "encryptionkey@0"
I20260812 02:59:56.555377 15671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:56.599575 15671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:56.605922 15671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:56.606429 15671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:56.609316 15671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:56.613634 15671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:56.613906 15671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:56.614194 15671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:56.614387 15671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:56.765203 15671 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:40527
I20260812 02:59:56.765307 15927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:40527 every 8 connection(s)
I20260812 02:59:56.767993 15671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 02:59:56.772889 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 15671
I20260812 02:59:56.773617 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 02:59:56.775702 10713 external_mini_cluster.cc:1428] Setting key 24df91f776fe54088bf758bc24c72bb3
Aug 12 02:59:56 dist-test-slave-zpvx krb5kdc[15451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503596, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 02:59:56.807617 15936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:56.772322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:51375 (local address 127.10.118.126:33183)
0812 02:59:56.774934 (+ 2612us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:56.774967 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:56.778763 (+ 3796us) server_negotiation.cc:407] Connection header received
0812 02:59:56.783633 (+ 4870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:56.783668 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:56.783881 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:56.784070 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:56.786408 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:56.787329 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:56.789596 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:56.790181 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:56.790498 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:56.796592 (+ 6094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:56.796616 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:56.796632 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:56.796726 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:56.800603 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:56.801770 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:56.801794 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:56.801811 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:56.802012 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:56.802764 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:56.802784 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:56.802800 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:56.803360 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:56.803612 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:56.804789 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0812 02:59:56.805286 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2004,"thread_start_us":1664,"threads_started":1}
I20260812 02:59:56.808585 15938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:56.772722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:33183 (local address 127.10.118.65:51375)
0812 02:59:56.777607 (+ 4885us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:56.777748 (+ 141us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:56.783279 (+ 5531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:56.784353 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:56.784413 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:56.785264 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:56.786117 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:56.786159 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:56.787588 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:56.787609 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:56.789318 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:56.789360 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:56.790537 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:56.790573 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:56.791039 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:56.791998 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:56.792109 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:56.796226 (+ 4117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:56.800822 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:56.800862 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:56.800884 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:56.801551 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:56.802240 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:56.802274 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:56.802292 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:56.802549 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:56.803572 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:56.803613 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:56.804058 (+ 445us) client_negotiation.cc:769] Sending connection context
0812 02:59:56.804516 (+ 458us) client_negotiation.cc:240] Negotiation successful
0812 02:59:56.805394 (+ 878us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4156,"thread_start_us":3656,"threads_started":1}
I20260812 02:59:56.810591 15931 heartbeater.cc:347] Connected to a master server at 127.10.118.126:33183
I20260812 02:59:56.810997 15931 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:56.811949 15931 heartbeater.cc:511] Master 127.10.118.126:33183 requested a full tablet report, sending...
I20260812 02:59:56.814569 15545 ts_manager.cc:194] Registered new tserver with Master: 67c8c2c1636345a4b47674e5f0a59fd5 (127.10.118.65:40527)
I20260812 02:59:56.816735 15545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:51375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 02:59:56.834862 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:33183
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:57.144935 15943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:57.145483 15943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:57.146144 15943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:57.176518 15943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:57.176901 15943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:57.177946 15943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 02:59:57.212862 15943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:57.214037 15943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:57.215082 15943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:57.233206 15959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:57.820474 15931 heartbeater.cc:503] Master 127.10.118.126:33183 was elected leader, sending a full tablet report...
W20260812 02:59:57.233201 15955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:57.235241 15943 server_base.cc:1034] running on GCE node
W20260812 02:59:57.233386 15957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:58.430639 15943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 02:59:58.433202 15943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 02:59:58.434577 15943 hybrid_clock.cc:648] HybridClock initialized: now 1786503598434533 us; error 62 us; skew 500 ppm
I20260812 02:59:58.435508 15943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 02:59:58 dist-test-slave-zpvx krb5kdc[15451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503598, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 02:59:58.441933 15943 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 02:59:58.448550 15943 webserver.cc:460] Webserver started at http://127.10.118.66:38399/ using document root <none> and password file <none>
I20260812 02:59:58.449554 15943 fs_manager.cc:359] Metadata directory not provided
I20260812 02:59:58.449822 15943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 02:59:58.450345 15943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 02:59:58.456144 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "f813c072ffdf45c4ac485da4ec4a44c5"
format_stamp: "Formatted at 2026-08-12 02:59:58 on dist-test-slave-zpvx"
server_key: "3b1bc3d7f7d3093b83b5f911fa8913d0"
server_key_iv: "e47be9f98002e1481cec0cbe6a4169ac"
server_key_version: "encryptionkey@0"
I20260812 02:59:58.457456 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "f813c072ffdf45c4ac485da4ec4a44c5"
format_stamp: "Formatted at 2026-08-12 02:59:58 on dist-test-slave-zpvx"
server_key: "3b1bc3d7f7d3093b83b5f911fa8913d0"
server_key_iv: "e47be9f98002e1481cec0cbe6a4169ac"
server_key_version: "encryptionkey@0"
I20260812 02:59:58.465201 15943 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260812 02:59:58.471119 15975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:58.472301 15943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 02:59:58.472666 15943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "f813c072ffdf45c4ac485da4ec4a44c5"
format_stamp: "Formatted at 2026-08-12 02:59:58 on dist-test-slave-zpvx"
server_key: "3b1bc3d7f7d3093b83b5f911fa8913d0"
server_key_iv: "e47be9f98002e1481cec0cbe6a4169ac"
server_key_version: "encryptionkey@0"
I20260812 02:59:58.473033 15943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 02:59:58.524506 15943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 02:59:58.530936 15943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 02:59:58.531430 15943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 02:59:58.534412 15943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 02:59:58.538815 15943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 02:59:58.539019 15943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:58.539368 15943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 02:59:58.539546 15943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 02:59:58.687490 15943 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:37775
I20260812 02:59:58.687654 16204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:37775 every 8 connection(s)
I20260812 02:59:58.691745 15943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 02:59:58.692631 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 15943
I20260812 02:59:58.693220 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 02:59:58.694649 10713 external_mini_cluster.cc:1428] Setting key 1131e9fdddf92311a99fd33bd0a339fa
Aug 12 02:59:58 dist-test-slave-zpvx krb5kdc[15451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503598, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 02:59:58.734195 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:58.698612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:33183 (local address 127.10.118.66:60767)
0812 02:59:58.701806 (+ 3194us) negotiation.cc:107] Waiting for socket to connect
0812 02:59:58.701874 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 02:59:58.706863 (+ 4989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 02:59:58.707899 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 02:59:58.707936 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 02:59:58.709112 (+ 1176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 02:59:58.710102 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:58.710144 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:58.711638 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:58.711687 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:58.714857 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 02:59:58.714923 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 02:59:58.716083 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 02:59:58.716136 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 02:59:58.716897 (+ 761us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:58.717980 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 02:59:58.718088 (+ 108us) client_negotiation.cc:656] calling sasl_client_start()
0812 02:59:58.722151 (+ 4063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 02:59:58.726749 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:58.726779 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:58.726816 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:58.727561 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:58.728340 (+ 779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 02:59:58.728406 (+ 66us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 02:59:58.728442 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0812 02:59:58.728781 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 02:59:58.730207 (+ 1426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 02:59:58.730250 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 02:59:58.730785 (+ 535us) client_negotiation.cc:769] Sending connection context
0812 02:59:58.731124 (+ 339us) client_negotiation.cc:240] Negotiation successful
0812 02:59:58.731655 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2603,"thread_start_us":1804,"threads_started":1}
I20260812 02:59:58.734453 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 02:59:58.700529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:60767 (local address 127.10.118.126:33183)
0812 02:59:58.705357 (+ 4828us) server_negotiation.cc:206] Beginning negotiation
0812 02:59:58.705379 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 02:59:58.705430 (+ 51us) server_negotiation.cc:407] Connection header received
0812 02:59:58.707201 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 02:59:58.707235 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 02:59:58.707433 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 02:59:58.707613 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 02:59:58.710388 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:58.711342 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:58.715142 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 02:59:58.715721 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 02:59:58.715920 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 02:59:58.722504 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 02:59:58.722524 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 02:59:58.722540 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 02:59:58.722639 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0812 02:59:58.726537 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:58.727772 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:58.727821 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:58.727838 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:58.728070 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 02:59:58.729034 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 02:59:58.729100 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 02:59:58.729133 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 02:59:58.729948 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 02:59:58.730224 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 02:59:58.731448 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0812 02:59:58.731962 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4382,"thread_start_us":4110,"threads_started":1}
I20260812 02:59:58.736852 16205 heartbeater.cc:347] Connected to a master server at 127.10.118.126:33183
I20260812 02:59:58.737370 16205 heartbeater.cc:464] Registering TS with master...
I20260812 02:59:58.738329 16205 heartbeater.cc:511] Master 127.10.118.126:33183 requested a full tablet report, sending...
I20260812 02:59:58.740724 15545 ts_manager.cc:194] Registered new tserver with Master: f813c072ffdf45c4ac485da4ec4a44c5 (127.10.118.66:37775)
I20260812 02:59:58.742260 15545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:60767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 02:59:58.758168 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:33183
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:42809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 02:59:59.072166 16217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 02:59:59.072723 16217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 02:59:59.073421 16217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 02:59:59.104267 16217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 02:59:59.104642 16217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 02:59:59.105598 16217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
W20260812 02:59:59.109876 15618 debug-util.cc:398] Leaking SignalData structure 0x7b08000a6aa0 after lost signal to thread 15480
I20260812 02:59:59.142215 16217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:42809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:33183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 02:59:59.143419 16217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 02:59:59.144510 16217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 02:59:59.164775 16232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:59.746331 16205 heartbeater.cc:503] Master 127.10.118.126:33183 was elected leader, sending a full tablet report...
W20260812 02:59:59.165108 16237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 02:59:59.167536 16217 server_base.cc:1034] running on GCE node
W20260812 02:59:59.166760 16231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:00.358794 16217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:00.361871 16217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:00.363364 16217 hybrid_clock.cc:648] HybridClock initialized: now 1786503600363300 us; error 82 us; skew 500 ppm
I20260812 03:00:00.364322 16217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:00 dist-test-slave-zpvx krb5kdc[15451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:00.371701 16217 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:00:00.377928 16217 webserver.cc:460] Webserver started at http://127.10.118.67:35467/ using document root <none> and password file <none>
I20260812 03:00:00.378890 16217 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:00.379134 16217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:00.379638 16217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:00.385185 16217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "f62722b169154f57a1f8372327eefa01"
format_stamp: "Formatted at 2026-08-12 03:00:00 on dist-test-slave-zpvx"
server_key: "023859f9a12d4fc72f96f5b3eb322593"
server_key_iv: "f024d294bbda75d13deaafe46ab7dbe2"
server_key_version: "encryptionkey@0"
I20260812 03:00:00.386396 16217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "f62722b169154f57a1f8372327eefa01"
format_stamp: "Formatted at 2026-08-12 03:00:00 on dist-test-slave-zpvx"
server_key: "023859f9a12d4fc72f96f5b3eb322593"
server_key_iv: "f024d294bbda75d13deaafe46ab7dbe2"
server_key_version: "encryptionkey@0"
I20260812 03:00:00.394963 16217 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260812 03:00:00.401296 16254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:00.402508 16217 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 03:00:00.402856 16217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "f62722b169154f57a1f8372327eefa01"
format_stamp: "Formatted at 2026-08-12 03:00:00 on dist-test-slave-zpvx"
server_key: "023859f9a12d4fc72f96f5b3eb322593"
server_key_iv: "f024d294bbda75d13deaafe46ab7dbe2"
server_key_version: "encryptionkey@0"
I20260812 03:00:00.403261 16217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:00.455372 16217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:00.461531 16217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:00.462021 16217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:00.465279 16217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:00.469714 16217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:00.469944 16217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:00.470213 16217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:00.470398 16217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:00.649667 16217 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:44269
I20260812 03:00:00.649798 16465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:44269 every 8 connection(s)
I20260812 03:00:00.652534 16217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:00:00.655886 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 16217
I20260812 03:00:00.656622 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:00:00.658396 10713 external_mini_cluster.cc:1428] Setting key 281273d38b0765ed05bcdf99c1180fb9
Aug 12 03:00:00 dist-test-slave-zpvx krb5kdc[15451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503600, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:00.715797 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:00.657836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:33183 (local address 127.10.118.67:38367)
0812 03:00:00.675906 (+ 18070us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:00.675983 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:00.682167 (+ 6184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:00.683516 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:00.683551 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:00.684926 (+ 1375us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:00.686280 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:00.686341 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:00.688702 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:00.688734 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:00.691499 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:00.691556 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:00.693813 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:00.693850 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:00.694495 (+ 645us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:00.696039 (+ 1544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:00.696151 (+ 112us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:00.700772 (+ 4621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:00.707671 (+ 6899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:00.707696 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:00.707725 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:00.708599 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:00.709429 (+ 830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:00.709451 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:00.709465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:00.709743 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:00.711047 (+ 1304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:00.711104 (+ 57us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:00.711659 (+ 555us) client_negotiation.cc:769] Sending connection context
0812 03:00:00.712136 (+ 477us) client_negotiation.cc:240] Negotiation successful
0812 03:00:00.712932 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17612,"thread_start_us":16128,"threads_started":1}
I20260812 03:00:00.716734 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:00.659845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:38367 (local address 127.10.118.126:33183)
0812 03:00:00.673648 (+ 13803us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:00.673677 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:00.676160 (+ 2483us) server_negotiation.cc:407] Connection header received
0812 03:00:00.682560 (+ 6400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:00.682596 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:00.682889 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:00.683094 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:00.686686 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:00.688282 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:00.691835 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:00.692909 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:00.693167 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:00.701213 (+ 8046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:00.701250 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:00.701281 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:00.701440 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:00.707419 (+ 5979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:00.708828 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:00.708874 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:00.708904 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:00.709208 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:00.709945 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:00.709979 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:00.710006 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:00.710781 (+ 775us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:00.711041 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:00.712788 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0812 03:00:00.713357 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13278,"thread_start_us":12966,"threads_started":1}
I20260812 03:00:00.718482 16466 heartbeater.cc:347] Connected to a master server at 127.10.118.126:33183
I20260812 03:00:00.718914 16466 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:00.720098 16466 heartbeater.cc:511] Master 127.10.118.126:33183 requested a full tablet report, sending...
I20260812 03:00:00.722498 15545 ts_manager.cc:194] Registered new tserver with Master: f62722b169154f57a1f8372327eefa01 (127.10.118.67:44269)
I20260812 03:00:00.724131 15545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:38367
I20260812 03:00:00.730652 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 03:00:00.789754 16471 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:00:00.783574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.10.118.126:33183)
0812 03:00:00.784284 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:00.784322 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:00.785447 (+ 1125us) server_negotiation.cc:407] Connection header received
0812 03:00:00.785884 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:00.785903 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:00.786079 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:00.786262 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:00.788733 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49012: BlockingRecv error: recv got EOF from 127.0.0.1:49012 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 03:00:00.797447 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 15671
I20260812 03:00:00.854295 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 15943
I20260812 03:00:00.896245 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 16217
I20260812 03:00:00.925915 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 15478
2026-08-12T03:00:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (9658 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:00:01 dist-test-slave-zpvx krb5kdc[16494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:00:01 dist-test-slave-zpvx krb5kdc[16494](info): set up 2 sockets
Aug 12 03:00:01 dist-test-slave-zpvx krb5kdc[16494](info): commencing operation
krb5kdc: starting...
W20260812 03:00:02.332136 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:00:02 dist-test-slave-zpvx krb5kdc[16494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:00:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:00:02.574065 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44425
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:44425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:02.886801 16521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:02.887456 16521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:02.888084 16521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:02.919032 16521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:02.919416 16521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:02.919765 16521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:02.920035 16521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:02.920312 16521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:00:02.958673 16521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:44425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:02.959903 16521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:02.961037 16521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:02.978611 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:02.978626 16537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:02.979509 16536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:02.980731 16521 server_base.cc:1034] running on GCE node
I20260812 03:00:04.212525 16521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:04.215188 16521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:04.216485 16521 hybrid_clock.cc:648] HybridClock initialized: now 1786503604216443 us; error 61 us; skew 500 ppm
I20260812 03:00:04.217322 16521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:04 dist-test-slave-zpvx krb5kdc[16494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503604, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:04.223457 16521 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:00:04.228768 16521 webserver.cc:460] Webserver started at http://127.10.118.126:35737/ using document root <none> and password file <none>
I20260812 03:00:04.229703 16521 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:04.229934 16521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:04.230409 16521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:04.236095 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "d0f345a135a54b3294c946c619aa25d7"
format_stamp: "Formatted at 2026-08-12 03:00:04 on dist-test-slave-zpvx"
server_key: "f20081dfbdd6fca0ae2be38cc830aff4"
server_key_iv: "d387d4a57fad22d1b8085d90d1f276a9"
server_key_version: "encryptionkey@0"
I20260812 03:00:04.237375 16521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "d0f345a135a54b3294c946c619aa25d7"
format_stamp: "Formatted at 2026-08-12 03:00:04 on dist-test-slave-zpvx"
server_key: "f20081dfbdd6fca0ae2be38cc830aff4"
server_key_iv: "d387d4a57fad22d1b8085d90d1f276a9"
server_key_version: "encryptionkey@0"
I20260812 03:00:04.244800 16521 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:00:04.250569 16560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:04.251631 16521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:00:04.251972 16521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "d0f345a135a54b3294c946c619aa25d7"
format_stamp: "Formatted at 2026-08-12 03:00:04 on dist-test-slave-zpvx"
server_key: "f20081dfbdd6fca0ae2be38cc830aff4"
server_key_iv: "d387d4a57fad22d1b8085d90d1f276a9"
server_key_version: "encryptionkey@0"
I20260812 03:00:04.252418 16521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:04.296597 16521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:04.302548 16521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:04.303041 16521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:04.373526 16521 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:44425
I20260812 03:00:04.373611 16659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:44425 every 8 connection(s)
I20260812 03:00:04.376348 16521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:04.381662 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:04.385622 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 16521
I20260812 03:00:04.386322 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:04.387480 10713 external_mini_cluster.cc:1428] Setting key d82aabf597fcd68a8401c9a6e21a85de
I20260812 03:00:04.404898 16661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7: Bootstrap starting.
I20260812 03:00:04.412065 16661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7: Neither blocks nor log segments found. Creating new log.
Aug 12 03:00:04 dist-test-slave-zpvx krb5kdc[16494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:04.413933 16661 log.cc:824] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:04.419673 16661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7: No bootstrap required, opened a new log
I20260812 03:00:04.425473 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:04.390055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.10.118.126:44425)
0812 03:00:04.392858 (+ 2803us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:04.392885 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:04.392944 (+ 59us) server_negotiation.cc:407] Connection header received
0812 03:00:04.396897 (+ 3953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:04.396935 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:04.397333 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:04.397669 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:04.401290 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:04.402937 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:04.405836 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:04.406711 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:04.406954 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:04.414058 (+ 7104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:04.414119 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:04.414148 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:04.414321 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:04.418703 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:04.419829 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:04.419852 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:04.419893 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:04.420183 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:04.420891 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:04.420911 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:04.420925 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:04.421648 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:04.422136 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:04.422869 (+ 733us) server_negotiation.cc:299] Negotiation successful
0812 03:00:04.423520 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2366,"thread_start_us":1839,"threads_started":1}
I20260812 03:00:04.430258 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:00:04.440471 16661 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f345a135a54b3294c946c619aa25d7" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44425 } }
I20260812 03:00:04.441191 16661 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:04.441437 16661 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f345a135a54b3294c946c619aa25d7, State: Initialized, Role: FOLLOWER
I20260812 03:00:04.442198 16661 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f345a135a54b3294c946c619aa25d7" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44425 } }
I20260812 03:00:04.442749 16661 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:04.443029 16661 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:04.443338 16661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:04.447433 16661 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f345a135a54b3294c946c619aa25d7" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44425 } }
I20260812 03:00:04.448210 16661 leader_election.cc:302] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0f345a135a54b3294c946c619aa25d7; no voters:
I20260812 03:00:04.450385 16661 leader_election.cc:288] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:04.450754 16673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:04.454197 16673 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [term 1 LEADER]: Becoming Leader. State: Replica: d0f345a135a54b3294c946c619aa25d7, State: Running, Role: LEADER
I20260812 03:00:04.455305 16673 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f345a135a54b3294c946c619aa25d7" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44425 } }
I20260812 03:00:04.455953 16661 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:04.472815 16676 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f345a135a54b3294c946c619aa25d7. Latest consensus state: current_term: 1 leader_uuid: "d0f345a135a54b3294c946c619aa25d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f345a135a54b3294c946c619aa25d7" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44425 } } }
I20260812 03:00:04.473915 16676 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:04.473826 16675 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f345a135a54b3294c946c619aa25d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0f345a135a54b3294c946c619aa25d7" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44425 } } }
I20260812 03:00:04.474498 16675 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:04.483848 16692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:04.495405 16692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:04.524578 16692 catalog_manager.cc:1347] Generated new cluster ID: bcf0c2e97a1d45888d6c15f775264ad2
I20260812 03:00:04.524892 16692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:04.546885 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:04.520766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.10.118.126:44425)
0812 03:00:04.521363 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:04.521389 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:04.524334 (+ 2945us) server_negotiation.cc:407] Connection header received
0812 03:00:04.525106 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:04.525153 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:04.525496 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:04.525707 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:04.529624 (+ 3917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:04.531497 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:04.535409 (+ 3912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:04.536852 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:04.537154 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:04.540249 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:04.540311 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:04.540351 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:04.541947 (+ 1596us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:04.542113 (+ 166us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:04.542221 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:04.542449 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:04.544067 (+ 1618us) server_negotiation.cc:299] Negotiation successful
0812 03:00:04.544410 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":163}
I20260812 03:00:04.578756 16692 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:04.581131 16692 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:04.595552 16692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f345a135a54b3294c946c619aa25d7: Generated new TSK 0
I20260812 03:00:04.596503 16692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:00:04.608507 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 16521
2026-08-12T03:00:04Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3673 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:00:04 dist-test-slave-zpvx krb5kdc[16732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:00:04 dist-test-slave-zpvx krb5kdc[16732](info): set up 2 sockets
Aug 12 03:00:04 dist-test-slave-zpvx krb5kdc[16732](info): commencing operation
krb5kdc: starting...
W20260812 03:00:06.004624 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:00:06 dist-test-slave-zpvx krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:00:06.238054 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:34923
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:34923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:06.557334 16760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:06.557986 16760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:06.558627 16760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:06.589074 16760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:06.589452 16760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:06.589787 16760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:06.590051 16760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:06.590317 16760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:00:06.627034 16760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:34923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:34923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:06.628258 16760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:06.629379 16760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:06.648531 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:06.649966 16760 server_base.cc:1034] running on GCE node
W20260812 03:00:06.648761 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:06.648545 16772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:07.859493 16760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:07.862833 16760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:07.864298 16760 hybrid_clock.cc:648] HybridClock initialized: now 1786503607864247 us; error 72 us; skew 500 ppm
I20260812 03:00:07.865216 16760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:07 dist-test-slave-zpvx krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503607, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:07.871487 16760 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:00:07.877068 16760 webserver.cc:460] Webserver started at http://127.10.118.126:41983/ using document root <none> and password file <none>
I20260812 03:00:07.878067 16760 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:07.878316 16760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:07.878813 16760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:07.884291 16760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "698be31ba5864885845554479487ff49"
format_stamp: "Formatted at 2026-08-12 03:00:07 on dist-test-slave-zpvx"
server_key: "1246705497204462bec5ed0e13fe559d"
server_key_iv: "433c48df81a40023f7dc273f57fc93e4"
server_key_version: "encryptionkey@0"
I20260812 03:00:07.885572 16760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "698be31ba5864885845554479487ff49"
format_stamp: "Formatted at 2026-08-12 03:00:07 on dist-test-slave-zpvx"
server_key: "1246705497204462bec5ed0e13fe559d"
server_key_iv: "433c48df81a40023f7dc273f57fc93e4"
server_key_version: "encryptionkey@0"
I20260812 03:00:07.892992 16760 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:00:07.899051 16801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:07.900082 16760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 03:00:07.900472 16760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "698be31ba5864885845554479487ff49"
format_stamp: "Formatted at 2026-08-12 03:00:07 on dist-test-slave-zpvx"
server_key: "1246705497204462bec5ed0e13fe559d"
server_key_iv: "433c48df81a40023f7dc273f57fc93e4"
server_key_version: "encryptionkey@0"
I20260812 03:00:07.900854 16760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:07.942937 16760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:07.949095 16760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:07.949599 16760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:08.020694 16760 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:34923
I20260812 03:00:08.020800 16880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:34923 every 8 connection(s)
I20260812 03:00:08.023595 16760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:08.029145 16886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:08.029094 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 16760
I20260812 03:00:08.029573 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:08.030777 10713 external_mini_cluster.cc:1428] Setting key 386c5a7ebd0a6e4894efc72439d47fb7
Aug 12 03:00:08 dist-test-slave-zpvx krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:08.056792 16886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49: Bootstrap starting.
I20260812 03:00:08.063993 16886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:08.065786 16886 log.cc:824] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:08.069911 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:08.033190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.10.118.126:34923)
0812 03:00:08.035773 (+ 2583us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:08.035812 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:08.035884 (+ 72us) server_negotiation.cc:407] Connection header received
0812 03:00:08.040275 (+ 4391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:08.040333 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:08.040860 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:08.041336 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:08.043544 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:08.046259 (+ 2715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:08.048173 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:08.049621 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:08.049986 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:08.055367 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:08.055405 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:08.055446 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:08.055589 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:08.061988 (+ 6399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:08.063161 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:08.063193 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:08.063222 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:08.063552 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:08.064470 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:08.064494 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:08.064513 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:08.065347 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:08.065909 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:08.066758 (+ 849us) server_negotiation.cc:299] Negotiation successful
0812 03:00:08.067505 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2041,"thread_start_us":1643,"threads_started":1}
I20260812 03:00:08.071810 16886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49: No bootstrap required, opened a new log
I20260812 03:00:08.091985 16886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698be31ba5864885845554479487ff49" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34923 } }
I20260812 03:00:08.092749 16886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:08.092953 16886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 698be31ba5864885845554479487ff49, State: Initialized, Role: FOLLOWER
I20260812 03:00:08.093587 16886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698be31ba5864885845554479487ff49" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34923 } }
I20260812 03:00:08.094066 16886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:08.094285 16886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:08.094527 16886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:08.098490 16886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698be31ba5864885845554479487ff49" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34923 } }
I20260812 03:00:08.099337 16886 leader_election.cc:302] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 698be31ba5864885845554479487ff49; no voters:
I20260812 03:00:08.101085 16886 leader_election.cc:288] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:08.101476 16898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:08.104055 16898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [term 1 LEADER]: Becoming Leader. State: Replica: 698be31ba5864885845554479487ff49, State: Running, Role: LEADER
I20260812 03:00:08.104939 16898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698be31ba5864885845554479487ff49" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34923 } }
I20260812 03:00:08.105829 16886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:08.116237 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 698be31ba5864885845554479487ff49. Latest consensus state: current_term: 1 leader_uuid: "698be31ba5864885845554479487ff49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698be31ba5864885845554479487ff49" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34923 } } }
I20260812 03:00:08.116991 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:08.118330 16901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "698be31ba5864885845554479487ff49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "698be31ba5864885845554479487ff49" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34923 } } }
I20260812 03:00:08.119164 16901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:08.121114 16918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:08.132562 16918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:08.149451 16918 catalog_manager.cc:1347] Generated new cluster ID: 4f1f9cba7127428b8edc2eb75b18bed0
I20260812 03:00:08.149775 16918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:08.175961 16918 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:08.178238 16918 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:08.194419 16918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 698be31ba5864885845554479487ff49: Generated new TSK 0
I20260812 03:00:08.195479 16918 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:00:08.266829 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34923
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:08.582880 16946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:08.583457 16946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:08.584105 16946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:08.615085 16946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:08.615489 16946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:08.616741 16946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:00:08.652633 16946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:08.653837 16946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:08.654875 16946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:08.675665 16959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:08.676383 16958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:08.678130 16969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:08.678215 16946 server_base.cc:1034] running on GCE node
I20260812 03:00:09.884485 16946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:09.886858 16946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:09.888200 16946 hybrid_clock.cc:648] HybridClock initialized: now 1786503609888163 us; error 56 us; skew 500 ppm
I20260812 03:00:09.889103 16946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:09 dist-test-slave-zpvx krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503609, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:09.901361 16946 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:00:09.907549 16946 webserver.cc:460] Webserver started at http://127.10.118.65:37203/ using document root <none> and password file <none>
I20260812 03:00:09.908542 16946 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:09.908866 16946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:09.909358 16946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:09.914650 16946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "c1d0695634524abeb58d08ff4aee96a7"
format_stamp: "Formatted at 2026-08-12 03:00:09 on dist-test-slave-zpvx"
server_key: "fe098083e48b2dbb55eeda4dbef805fd"
server_key_iv: "47f413f8ac01bd4527dff3ef46428047"
server_key_version: "encryptionkey@0"
I20260812 03:00:09.915886 16946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "c1d0695634524abeb58d08ff4aee96a7"
format_stamp: "Formatted at 2026-08-12 03:00:09 on dist-test-slave-zpvx"
server_key: "fe098083e48b2dbb55eeda4dbef805fd"
server_key_iv: "47f413f8ac01bd4527dff3ef46428047"
server_key_version: "encryptionkey@0"
I20260812 03:00:09.923893 16946 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 03:00:09.929673 16982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:09.930733 16946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 03:00:09.931102 16946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "c1d0695634524abeb58d08ff4aee96a7"
format_stamp: "Formatted at 2026-08-12 03:00:09 on dist-test-slave-zpvx"
server_key: "fe098083e48b2dbb55eeda4dbef805fd"
server_key_iv: "47f413f8ac01bd4527dff3ef46428047"
server_key_version: "encryptionkey@0"
I20260812 03:00:09.931504 16946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:09.973802 16946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:09.979303 16946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:09.979781 16946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:09.982335 16946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:09.986934 16946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:09.987277 16946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 03:00:09.987645 16946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:09.987946 16946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:10.127188 16946 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:38641
I20260812 03:00:10.127312 17185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:38641 every 8 connection(s)
I20260812 03:00:10.129843 16946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:00:10.140486 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 16946
I20260812 03:00:10.141070 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:00:10.142551 10713 external_mini_cluster.cc:1428] Setting key d423aaa9cea107917fc4f06794d22fd7
Aug 12 03:00:10 dist-test-slave-zpvx krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503609, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
I20260812 03:00:10.172367 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:10.134705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34923 (local address 127.10.118.65:57179)
0812 03:00:10.137790 (+ 3085us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:10.137857 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:10.143964 (+ 6107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:10.145107 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:10.145131 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:10.145933 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:10.147133 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:10.147236 (+ 103us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:10.148811 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:10.148846 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:10.151607 (+ 2761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:10.151651 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:10.153321 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:10.153349 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:10.153713 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:10.154676 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:10.154747 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:10.158521 (+ 3774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:10.165425 (+ 6904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:10.165448 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:10.165475 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:10.166246 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:10.167100 (+ 854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:10.167129 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:10.167143 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:10.167436 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:10.168960 (+ 1524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:10.168989 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:10.169435 (+ 446us) client_negotiation.cc:769] Sending connection context
0812 03:00:10.169768 (+ 333us) client_negotiation.cc:240] Negotiation successful
0812 03:00:10.170295 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2579,"thread_start_us":1802,"threads_started":1}
I20260812 03:00:10.173007 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:10.134363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:57179 (local address 127.10.118.126:34923)
0812 03:00:10.135684 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:10.135707 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:10.138013 (+ 2306us) server_negotiation.cc:407] Connection header received
0812 03:00:10.144360 (+ 6347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:10.144435 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:10.144679 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:10.144843 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:10.147499 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:10.148550 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:10.151866 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:10.152960 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:10.153311 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:10.158899 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:10.158949 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:10.158989 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:10.159168 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:10.165166 (+ 5998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:10.166505 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:10.166545 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:10.166575 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:10.166888 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:10.167665 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:10.167709 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:10.167742 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:10.168730 (+ 988us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:10.169010 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:10.170097 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0812 03:00:10.170690 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":639,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:00:10.174942 17186 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34923
I20260812 03:00:10.175362 17186 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:10.176292 17186 heartbeater.cc:511] Master 127.10.118.126:34923 requested a full tablet report, sending...
I20260812 03:00:10.179280 16822 ts_manager.cc:194] Registered new tserver with Master: c1d0695634524abeb58d08ff4aee96a7 (127.10.118.65:38641)
I20260812 03:00:10.182559 16822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:57179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:00:10.204135 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34923
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:10.513902 17197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:10.514454 17197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:10.515110 17197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:10.545642 17197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:10.546007 17197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:10.546979 17197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:00:10.581295 17197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:10.582499 17197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:10.583546 17197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:10.601693 17210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:11.186466 17186 heartbeater.cc:503] Master 127.10.118.126:34923 was elected leader, sending a full tablet report...
W20260812 03:00:10.601788 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:10.603845 17208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:11.802526 17212 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:00:11.802661 17197 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:00:11.806234 17197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:11.808563 17197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:11.809950 17197 hybrid_clock.cc:648] HybridClock initialized: now 1786503611809915 us; error 48 us; skew 500 ppm
I20260812 03:00:11.810873 17197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:11 dist-test-slave-zpvx krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:11.817394 17197 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:00:11.822754 17197 webserver.cc:460] Webserver started at http://127.10.118.66:40381/ using document root <none> and password file <none>
I20260812 03:00:11.823674 17197 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:11.823884 17197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:11.824361 17197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:11.829660 17197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "33418dad2be0409db7a87518924c4692"
format_stamp: "Formatted at 2026-08-12 03:00:11 on dist-test-slave-zpvx"
server_key: "1c708b40f11fe371f1b7be5253eb2564"
server_key_iv: "9974421afc3b0fb8725d3cab723b3bab"
server_key_version: "encryptionkey@0"
I20260812 03:00:11.830832 17197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "33418dad2be0409db7a87518924c4692"
format_stamp: "Formatted at 2026-08-12 03:00:11 on dist-test-slave-zpvx"
server_key: "1c708b40f11fe371f1b7be5253eb2564"
server_key_iv: "9974421afc3b0fb8725d3cab723b3bab"
server_key_version: "encryptionkey@0"
I20260812 03:00:11.838306 17197 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260812 03:00:11.843888 17223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:11.844985 17197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 03:00:11.845407 17197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "33418dad2be0409db7a87518924c4692"
format_stamp: "Formatted at 2026-08-12 03:00:11 on dist-test-slave-zpvx"
server_key: "1c708b40f11fe371f1b7be5253eb2564"
server_key_iv: "9974421afc3b0fb8725d3cab723b3bab"
server_key_version: "encryptionkey@0"
I20260812 03:00:11.845803 17197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:11.883096 17197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:11.889070 17197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:11.889532 17197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:11.892091 17197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:11.896121 17197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:11.896443 17197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 03:00:11.896703 17197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:11.896875 17197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:12.038214 17197 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:35447
I20260812 03:00:12.038381 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:35447 every 8 connection(s)
I20260812 03:00:12.040894 17197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:00:12.049821 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 17197
I20260812 03:00:12.050355 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:00:12.051406 10713 external_mini_cluster.cc:1428] Setting key 365aa16adb35c95bdb9d947879c10f4e
Aug 12 03:00:12 dist-test-slave-zpvx krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503611, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:00:12.077862 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:12.045134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:59025 (local address 127.10.118.126:34923)
0812 03:00:12.046257 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:12.046272 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:12.049062 (+ 2790us) server_negotiation.cc:407] Connection header received
0812 03:00:12.054675 (+ 5613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:12.054696 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:12.054885 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:12.055099 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:12.057636 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:12.058573 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:12.060778 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:12.061392 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:12.061676 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:12.067069 (+ 5393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:12.067096 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:12.067112 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:12.067277 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:12.071031 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:12.072251 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:12.072272 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:12.072290 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:12.072545 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:12.073257 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:12.073276 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:12.073289 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:12.073791 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:12.074041 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:12.075091 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0812 03:00:12.075574 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":507,"threads_started":1}
I20260812 03:00:12.078630 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:12.045585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34923 (local address 127.10.118.66:59025)
0812 03:00:12.048824 (+ 3239us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:12.048893 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:12.054272 (+ 5379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:12.055399 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:12.055437 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:12.056342 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:12.057344 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:12.057389 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:12.058833 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:12.058853 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:12.060522 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:12.060561 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:12.061733 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:12.061771 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:12.062219 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:12.063144 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:12.063221 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:12.066730 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:12.071281 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:12.071322 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:12.071346 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:12.072031 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:12.072735 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:12.072770 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:12.072784 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:12.073047 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:12.074003 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:12.074042 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:12.074504 (+ 462us) client_negotiation.cc:769] Sending connection context
0812 03:00:12.074867 (+ 363us) client_negotiation.cc:240] Negotiation successful
0812 03:00:12.075578 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2646,"thread_start_us":1787,"threads_started":1}
I20260812 03:00:12.080792 17431 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34923
I20260812 03:00:12.081202 17431 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:12.082139 17431 heartbeater.cc:511] Master 127.10.118.126:34923 requested a full tablet report, sending...
I20260812 03:00:12.084504 16822 ts_manager.cc:194] Registered new tserver with Master: 33418dad2be0409db7a87518924c4692 (127.10.118.66:35447)
I20260812 03:00:12.086584 16822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:59025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:00:12.106436 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34923
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:12.416937 17439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:12.417472 17439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:12.418121 17439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:12.449272 17439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:12.449656 17439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:12.450686 17439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:00:12.487682 17439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:12.489167 17439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:12.490173 17439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:12.508999 17448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:13.090054 17431 heartbeater.cc:503] Master 127.10.118.126:34923 was elected leader, sending a full tablet report...
W20260812 03:00:12.509078 17449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:12.510803 17439 server_base.cc:1034] running on GCE node
W20260812 03:00:12.509970 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:13.696697 17439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:13.698994 17439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:13.700349 17439 hybrid_clock.cc:648] HybridClock initialized: now 1786503613700313 us; error 51 us; skew 500 ppm
I20260812 03:00:13.701231 17439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:13 dist-test-slave-zpvx krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503613, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:13.707664 17439 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:00:13.713001 17439 webserver.cc:460] Webserver started at http://127.10.118.67:35579/ using document root <none> and password file <none>
I20260812 03:00:13.713971 17439 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:13.714216 17439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:13.714684 17439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:13.720118 17439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "68aa409bcfa54600ab51c444ec104834"
format_stamp: "Formatted at 2026-08-12 03:00:13 on dist-test-slave-zpvx"
server_key: "21ca73263e971bbed75d20dbbb1a3a95"
server_key_iv: "22d77d4ba551e4f5434d41fb83752d31"
server_key_version: "encryptionkey@0"
I20260812 03:00:13.721555 17439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "68aa409bcfa54600ab51c444ec104834"
format_stamp: "Formatted at 2026-08-12 03:00:13 on dist-test-slave-zpvx"
server_key: "21ca73263e971bbed75d20dbbb1a3a95"
server_key_iv: "22d77d4ba551e4f5434d41fb83752d31"
server_key_version: "encryptionkey@0"
I20260812 03:00:13.729192 17439 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:00:13.734843 17470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:13.735881 17439 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:00:13.736212 17439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "68aa409bcfa54600ab51c444ec104834"
format_stamp: "Formatted at 2026-08-12 03:00:13 on dist-test-slave-zpvx"
server_key: "21ca73263e971bbed75d20dbbb1a3a95"
server_key_iv: "22d77d4ba551e4f5434d41fb83752d31"
server_key_version: "encryptionkey@0"
I20260812 03:00:13.736662 17439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:13.771692 17439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:13.777278 17439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:13.777760 17439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:13.780346 17439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:13.784464 17439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:13.784708 17439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:13.784956 17439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:13.785130 17439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:13.925273 17439 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:42441
I20260812 03:00:13.925384 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:42441 every 8 connection(s)
I20260812 03:00:13.927951 17439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:00:13.932330 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 17439
I20260812 03:00:13.933212 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:00:13.935005 10713 external_mini_cluster.cc:1428] Setting key 0be0590c14bd3194fd770af1913010bf
Aug 12 03:00:13 dist-test-slave-zpvx krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503613, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:13.965302 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:13.932514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:50497 (local address 127.10.118.126:34923)
0812 03:00:13.933879 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:13.933893 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:13.935963 (+ 2070us) server_negotiation.cc:407] Connection header received
0812 03:00:13.942097 (+ 6134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:13.942131 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:13.942314 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:13.942486 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:13.944865 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:13.945834 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:13.947903 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:13.948599 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:13.948886 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:13.954318 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:13.954343 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:13.954359 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:13.954459 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:13.958496 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:13.959634 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:13.959657 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:13.959673 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:13.959861 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:13.960601 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:13.960620 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:13.960636 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:13.961136 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:13.961375 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:13.962456 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0812 03:00:13.962936 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":770,"threads_started":1}
I20260812 03:00:13.966038 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:13.932922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34923 (local address 127.10.118.67:50497)
0812 03:00:13.935627 (+ 2705us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:13.935775 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:13.941767 (+ 5992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:13.942768 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:13.942804 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:13.943630 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:13.944591 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:13.944641 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:13.946091 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:13.946118 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:13.947650 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:13.947696 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:13.948927 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:13.948962 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:13.949424 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:13.950292 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:13.950357 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:13.953993 (+ 3636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:13.958699 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:13.958733 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:13.958761 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:13.959424 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:13.960042 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:13.960069 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:13.960099 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:13.960353 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:13.961339 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:13.961383 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:13.961848 (+ 465us) client_negotiation.cc:769] Sending connection context
0812 03:00:13.962221 (+ 373us) client_negotiation.cc:240] Negotiation successful
0812 03:00:13.962946 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2245,"thread_start_us":1546,"threads_started":1}
I20260812 03:00:13.967772 17691 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34923
I20260812 03:00:13.968230 17691 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:13.969408 17691 heartbeater.cc:511] Master 127.10.118.126:34923 requested a full tablet report, sending...
I20260812 03:00:13.971496 16822 ts_manager.cc:194] Registered new tserver with Master: 68aa409bcfa54600ab51c444ec104834 (127.10.118.67:42441)
I20260812 03:00:13.973359 16822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:50497
I20260812 03:00:13.984041 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 03:00:14 dist-test-slave-zpvx krb5kdc[16732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503614, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 03:00:14 dist-test-slave-zpvx krb5kdc[16732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503614, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:14.070045 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:14.044231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.10.118.126:34923)
0812 03:00:14.044838 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:14.044858 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:14.046070 (+ 1212us) server_negotiation.cc:407] Connection header received
0812 03:00:14.046560 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:14.046587 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:14.046776 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:14.046959 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:14.049753 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:14.050864 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:14.053107 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:14.053762 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:14.054062 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:14.059954 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:14.059978 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:14.059995 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:14.060114 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:14.063854 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:14.065103 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:14.065125 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:14.065143 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:14.065353 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:14.066086 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:14.066123 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:14.066148 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:14.066738 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:14.067148 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:14.067890 (+ 742us) server_negotiation.cc:299] Negotiation successful
0812 03:00:14.068452 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
W20260812 03:00:14.071044 16822 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45884
I20260812 03:00:14.074905 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 16946
I20260812 03:00:14.111313 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 17197
I20260812 03:00:14.145289 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 17439
I20260812 03:00:14.175781 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 16760
2026-08-12T03:00:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (9570 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:00:14 dist-test-slave-zpvx krb5kdc[17717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:00:14 dist-test-slave-zpvx krb5kdc[17717](info): set up 2 sockets
Aug 12 03:00:14 dist-test-slave-zpvx krb5kdc[17717](info): commencing operation
krb5kdc: starting...
W20260812 03:00:15.560858 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:00:15 dist-test-slave-zpvx krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:00:15.780853 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38505
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:38505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:16.094791 17738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:16.095443 17738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:16.096057 17738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:16.127413 17738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:16.127779 17738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:16.128080 17738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:16.128331 17738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:16.128641 17738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:00:16.164561 17738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:38505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:16.165748 17738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:16.166803 17738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:16.184357 17750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:16.184506 17749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:16.186182 17738 server_base.cc:1034] running on GCE node
W20260812 03:00:16.184970 17756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:17.397351 17738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:17.400053 17738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:17.401398 17738 hybrid_clock.cc:648] HybridClock initialized: now 1786503617401355 us; error 60 us; skew 500 ppm
I20260812 03:00:17.402252 17738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:17 dist-test-slave-zpvx krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503617, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:17.414454 17738 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:00:17.420050 17738 webserver.cc:460] Webserver started at http://127.10.118.126:33051/ using document root <none> and password file <none>
I20260812 03:00:17.421092 17738 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:17.421331 17738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:17.421799 17738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:17.427134 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "224944b42e974172abd79d2d8f15ecc4"
format_stamp: "Formatted at 2026-08-12 03:00:17 on dist-test-slave-zpvx"
server_key: "c54df9add12a2017ce2613b434b53c81"
server_key_iv: "3310b495b6c81e2ec77fea150c7392fa"
server_key_version: "encryptionkey@0"
I20260812 03:00:17.428306 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "224944b42e974172abd79d2d8f15ecc4"
format_stamp: "Formatted at 2026-08-12 03:00:17 on dist-test-slave-zpvx"
server_key: "c54df9add12a2017ce2613b434b53c81"
server_key_iv: "3310b495b6c81e2ec77fea150c7392fa"
server_key_version: "encryptionkey@0"
I20260812 03:00:17.435738 17738 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 03:00:17.441241 17768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:17.442408 17738 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 03:00:17.442744 17738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "224944b42e974172abd79d2d8f15ecc4"
format_stamp: "Formatted at 2026-08-12 03:00:17 on dist-test-slave-zpvx"
server_key: "c54df9add12a2017ce2613b434b53c81"
server_key_iv: "3310b495b6c81e2ec77fea150c7392fa"
server_key_version: "encryptionkey@0"
I20260812 03:00:17.443138 17738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:17.483642 17738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:17.489662 17738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:17.490181 17738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:17.560253 17738 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:38505
I20260812 03:00:17.560353 17851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:38505 every 8 connection(s)
I20260812 03:00:17.563051 17738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:17.567114 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 17738
I20260812 03:00:17.567677 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:17.569341 10713 external_mini_cluster.cc:1428] Setting key ef67d387fb000a3de40c399e1e9f16ab
I20260812 03:00:17.569346 17852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 03:00:17 dist-test-slave-zpvx krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:17.596830 17852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4: Bootstrap starting.
I20260812 03:00:17.602062 17852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:17.603781 17852 log.cc:824] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:17.607546 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:17.572326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.10.118.126:38505)
0812 03:00:17.575150 (+ 2824us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:17.575178 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:17.575235 (+ 57us) server_negotiation.cc:407] Connection header received
0812 03:00:17.579655 (+ 4420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:17.579695 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:17.580061 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:17.580461 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:17.583802 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:17.585515 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:17.588144 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:17.589094 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:17.589345 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:17.596051 (+ 6706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:17.596112 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:17.596143 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:17.596252 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:17.600558 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:17.601734 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:17.601758 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:17.601778 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:17.602041 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:17.602762 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:17.602782 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:17.602796 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:17.603540 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:17.603995 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:17.604782 (+ 787us) server_negotiation.cc:299] Negotiation successful
0812 03:00:17.605478 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2385,"thread_start_us":1887,"threads_started":1}
I20260812 03:00:17.609731 17852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4: No bootstrap required, opened a new log
I20260812 03:00:17.629580 17852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224944b42e974172abd79d2d8f15ecc4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38505 } }
I20260812 03:00:17.630264 17852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:17.630560 17852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 224944b42e974172abd79d2d8f15ecc4, State: Initialized, Role: FOLLOWER
I20260812 03:00:17.631340 17852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224944b42e974172abd79d2d8f15ecc4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38505 } }
I20260812 03:00:17.631863 17852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:17.632179 17852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:17.632522 17852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:17.636603 17852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224944b42e974172abd79d2d8f15ecc4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38505 } }
I20260812 03:00:17.637267 17852 leader_election.cc:302] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 224944b42e974172abd79d2d8f15ecc4; no voters:
I20260812 03:00:17.639015 17852 leader_election.cc:288] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:17.639490 17860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:17.643432 17852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:17.643653 17860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [term 1 LEADER]: Becoming Leader. State: Replica: 224944b42e974172abd79d2d8f15ecc4, State: Running, Role: LEADER
I20260812 03:00:17.644498 17860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224944b42e974172abd79d2d8f15ecc4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38505 } }
I20260812 03:00:17.655195 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "224944b42e974172abd79d2d8f15ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224944b42e974172abd79d2d8f15ecc4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38505 } } }
I20260812 03:00:17.657366 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:17.656709 17865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 224944b42e974172abd79d2d8f15ecc4. Latest consensus state: current_term: 1 leader_uuid: "224944b42e974172abd79d2d8f15ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "224944b42e974172abd79d2d8f15ecc4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38505 } } }
I20260812 03:00:17.657837 17865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:17.661067 17879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:17.675925 17879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:17.692994 17879 catalog_manager.cc:1347] Generated new cluster ID: 321cdd1400354cd49a7fbb471fb413e7
I20260812 03:00:17.693306 17879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:17.737860 17879 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:17.739634 17879 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:17.761016 17879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 224944b42e974172abd79d2d8f15ecc4: Generated new TSK 0
I20260812 03:00:17.762030 17879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:00:17.825188 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38505
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:18.166872 17893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:18.167415 17893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:18.168054 17893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:18.199476 17893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:18.199856 17893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:18.200845 17893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:00:18.236820 17893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:18.238081 17893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:18.239125 17893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:18.260412 17904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:18.261684 17902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:18.264472 17893 server_base.cc:1034] running on GCE node
W20260812 03:00:18.265447 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:19.481737 17893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:19.484055 17893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:19.485399 17893 hybrid_clock.cc:648] HybridClock initialized: now 1786503619485342 us; error 61 us; skew 500 ppm
I20260812 03:00:19.486246 17893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:19 dist-test-slave-zpvx krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503619, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:19.492590 17893 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:00:19.497387 17893 webserver.cc:460] Webserver started at http://127.10.118.65:37955/ using document root <none> and password file <none>
I20260812 03:00:19.498324 17893 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:19.498548 17893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:19.499043 17893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:19.504616 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "1ae0b3582b6c404387843160f5d22d89"
format_stamp: "Formatted at 2026-08-12 03:00:19 on dist-test-slave-zpvx"
server_key: "d5311cb71100b31d5da05e4a2873886f"
server_key_iv: "1592b3ffc5ad400ac1016ddc48eebad6"
server_key_version: "encryptionkey@0"
I20260812 03:00:19.505820 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "1ae0b3582b6c404387843160f5d22d89"
format_stamp: "Formatted at 2026-08-12 03:00:19 on dist-test-slave-zpvx"
server_key: "d5311cb71100b31d5da05e4a2873886f"
server_key_iv: "1592b3ffc5ad400ac1016ddc48eebad6"
server_key_version: "encryptionkey@0"
I20260812 03:00:19.513336 17893 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260812 03:00:19.518779 17921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:19.519765 17893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 03:00:19.520092 17893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "1ae0b3582b6c404387843160f5d22d89"
format_stamp: "Formatted at 2026-08-12 03:00:19 on dist-test-slave-zpvx"
server_key: "d5311cb71100b31d5da05e4a2873886f"
server_key_iv: "1592b3ffc5ad400ac1016ddc48eebad6"
server_key_version: "encryptionkey@0"
I20260812 03:00:19.520515 17893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:19.558508 17893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:19.564570 17893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:19.565034 17893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:19.567631 17893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:19.571594 17893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:19.571802 17893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:19.572099 17893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:19.572259 17893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:19.707921 17893 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:46207
I20260812 03:00:19.708014 18127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:46207 every 8 connection(s)
I20260812 03:00:19.710593 17893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:00:19.718824 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 17893
I20260812 03:00:19.719436 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:00:19.721036 10713 external_mini_cluster.cc:1428] Setting key ff1b369d3b2a9937778a74600259a245
Aug 12 03:00:19 dist-test-slave-zpvx krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503619, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:00:19.752910 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:19.714833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:60317 (local address 127.10.118.126:38505)
0812 03:00:19.716388 (+ 1555us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:19.716417 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:19.718033 (+ 1616us) server_negotiation.cc:407] Connection header received
0812 03:00:19.724734 (+ 6701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:19.724770 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:19.725000 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:19.725192 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:19.728803 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:19.729718 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:19.733480 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:19.734059 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:19.734265 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:19.740098 (+ 5833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:19.740140 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:19.740176 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:19.740350 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:19.746090 (+ 5740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:19.747305 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:19.747341 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:19.747373 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:19.747644 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:19.748412 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:19.748445 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:19.748477 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:19.749145 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:19.749343 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:19.750493 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0812 03:00:19.750885 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":644,"threads_started":1}
I20260812 03:00:19.754006 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:19.715176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38505 (local address 127.10.118.65:60317)
0812 03:00:19.717800 (+ 2624us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:19.717868 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:19.724311 (+ 6443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:19.725535 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:19.725575 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:19.726909 (+ 1334us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:19.728477 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:19.728559 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:19.730006 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:19.730057 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:19.733181 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:19.733250 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:19.734436 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:19.734479 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:19.735179 (+ 700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:19.736223 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:19.736299 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:19.739656 (+ 3357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:19.746323 (+ 6667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:19.746347 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:19.746373 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:19.747057 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:19.747865 (+ 808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:19.747884 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:19.747898 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:19.748172 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:19.749380 (+ 1208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:19.749421 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:19.749924 (+ 503us) client_negotiation.cc:769] Sending connection context
0812 03:00:19.750296 (+ 372us) client_negotiation.cc:240] Negotiation successful
0812 03:00:19.750911 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2187,"thread_start_us":1492,"threads_started":1}
I20260812 03:00:19.756289 18128 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38505
I20260812 03:00:19.756763 18128 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:19.757726 18128 heartbeater.cc:511] Master 127.10.118.126:38505 requested a full tablet report, sending...
I20260812 03:00:19.760427 17801 ts_manager.cc:194] Registered new tserver with Master: 1ae0b3582b6c404387843160f5d22d89 (127.10.118.65:46207)
I20260812 03:00:19.763629 17801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:60317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:00:19.780967 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38505
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:20.094859 18138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:20.095414 18138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:20.096024 18138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:20.127223 18138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:20.127555 18138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:20.128477 18138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:00:20.164167 18138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:20.165347 18138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:20.166361 18138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:20.184844 18149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:20.767606 18128 heartbeater.cc:503] Master 127.10.118.126:38505 was elected leader, sending a full tablet report...
W20260812 03:00:20.184894 18147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:20.186331 18138 server_base.cc:1034] running on GCE node
W20260812 03:00:20.185315 18146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:21.366380 18138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:21.369408 18138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:21.370878 18138 hybrid_clock.cc:648] HybridClock initialized: now 1786503621370818 us; error 82 us; skew 500 ppm
I20260812 03:00:21.371698 18138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:21 dist-test-slave-zpvx krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503621, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:21.377959 18138 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:00:21.383965 18138 webserver.cc:460] Webserver started at http://127.10.118.66:43097/ using document root <none> and password file <none>
I20260812 03:00:21.385201 18138 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:21.385490 18138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:21.385995 18138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:21.391542 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "0815a691b85c494fb7e91e2f952a4647"
format_stamp: "Formatted at 2026-08-12 03:00:21 on dist-test-slave-zpvx"
server_key: "91401df6f8780ad3efd5e076f1c03c68"
server_key_iv: "64ded5812728e7fdfbd8d83167cb946c"
server_key_version: "encryptionkey@0"
I20260812 03:00:21.392896 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "0815a691b85c494fb7e91e2f952a4647"
format_stamp: "Formatted at 2026-08-12 03:00:21 on dist-test-slave-zpvx"
server_key: "91401df6f8780ad3efd5e076f1c03c68"
server_key_iv: "64ded5812728e7fdfbd8d83167cb946c"
server_key_version: "encryptionkey@0"
I20260812 03:00:21.400710 18138 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:00:21.406522 18161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:21.407675 18138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 03:00:21.408034 18138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "0815a691b85c494fb7e91e2f952a4647"
format_stamp: "Formatted at 2026-08-12 03:00:21 on dist-test-slave-zpvx"
server_key: "91401df6f8780ad3efd5e076f1c03c68"
server_key_iv: "64ded5812728e7fdfbd8d83167cb946c"
server_key_version: "encryptionkey@0"
I20260812 03:00:21.408480 18138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:21.455482 18138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:21.461797 18138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:21.462261 18138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:21.465091 18138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:21.469455 18138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:21.469667 18138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:21.469889 18138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:21.470021 18138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:21.615934 18138 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:46083
I20260812 03:00:21.616055 18361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:46083 every 8 connection(s)
I20260812 03:00:21.618686 18138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:00:21.621178 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 18138
I20260812 03:00:21.621845 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:00:21.623347 10713 external_mini_cluster.cc:1428] Setting key bb6a37dcd25220f9c5ffca5cdbea1642
Aug 12 03:00:21 dist-test-slave-zpvx krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503621, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:00:21.662163 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:21.623641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:38449 (local address 127.10.118.126:38505)
0812 03:00:21.625352 (+ 1711us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:21.625373 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:21.627715 (+ 2342us) server_negotiation.cc:407] Connection header received
0812 03:00:21.635509 (+ 7794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:21.635531 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:21.635734 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:21.635882 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:21.639110 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:21.640770 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:21.643053 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:21.644233 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:21.644608 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:21.650308 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:21.650348 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:21.650377 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:21.650541 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:21.654793 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:21.656593 (+ 1800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:21.656616 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:21.656630 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:21.656844 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:21.657797 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:21.657816 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:21.657830 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:21.658344 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:21.658522 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:21.660035 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0812 03:00:21.660547 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":499,"threads_started":1}
I20260812 03:00:21.664134 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:21.624023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38505 (local address 127.10.118.66:38449)
0812 03:00:21.627435 (+ 3412us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:21.627528 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:21.634991 (+ 7463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:21.636258 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:21.636303 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:21.637709 (+ 1406us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:21.638757 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:21.638802 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:21.641087 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:21.641108 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:21.642764 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:21.642805 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:21.644659 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:21.644687 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:21.645166 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:21.646185 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:21.646272 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:21.649876 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:21.655069 (+ 5193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:21.655108 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:21.655151 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:21.656344 (+ 1193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:21.657066 (+ 722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:21.657104 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:21.657136 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:21.657570 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:21.658594 (+ 1024us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:21.658642 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:21.659335 (+ 693us) client_negotiation.cc:769] Sending connection context
0812 03:00:21.659800 (+ 465us) client_negotiation.cc:240] Negotiation successful
0812 03:00:21.660732 (+ 932us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2783,"thread_start_us":1930,"threads_started":1}
I20260812 03:00:21.666235 18362 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38505
I20260812 03:00:21.666692 18362 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:21.667667 18362 heartbeater.cc:511] Master 127.10.118.126:38505 requested a full tablet report, sending...
I20260812 03:00:21.670058 17801 ts_manager.cc:194] Registered new tserver with Master: 0815a691b85c494fb7e91e2f952a4647 (127.10.118.66:46083)
I20260812 03:00:21.671533 17801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:38449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:00:21.688899 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38505
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:22.017130 18372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:22.017735 18372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:22.018414 18372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:22.052349 18372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:22.052822 18372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:22.053793 18372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:00:22.090802 18372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:22.092082 18372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:22.093214 18372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:22.112910 18382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:22.675666 18362 heartbeater.cc:503] Master 127.10.118.126:38505 was elected leader, sending a full tablet report...
W20260812 03:00:22.112924 18381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:22.113067 18389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:22.115818 18372 server_base.cc:1034] running on GCE node
I20260812 03:00:23.723989 18372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:23.726375 18372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:23.727725 18372 hybrid_clock.cc:648] HybridClock initialized: now 1786503623727687 us; error 58 us; skew 500 ppm
I20260812 03:00:23.728693 18372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:23 dist-test-slave-zpvx krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503623, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:23.738790 18372 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:00:23.745178 18372 webserver.cc:460] Webserver started at http://127.10.118.67:38547/ using document root <none> and password file <none>
I20260812 03:00:23.746253 18372 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:23.746517 18372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:23.747035 18372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:23.752942 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "a99f26c0884c4246b7d51de4f925bb66"
format_stamp: "Formatted at 2026-08-12 03:00:23 on dist-test-slave-zpvx"
server_key: "83b975b82d97a6310f4ed0be7553ca56"
server_key_iv: "e32cdf9d79b5fd125939beb539616e2f"
server_key_version: "encryptionkey@0"
I20260812 03:00:23.754231 18372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "a99f26c0884c4246b7d51de4f925bb66"
format_stamp: "Formatted at 2026-08-12 03:00:23 on dist-test-slave-zpvx"
server_key: "83b975b82d97a6310f4ed0be7553ca56"
server_key_iv: "e32cdf9d79b5fd125939beb539616e2f"
server_key_version: "encryptionkey@0"
I20260812 03:00:23.762468 18372 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260812 03:00:23.768983 18404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:23.770126 18372 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 03:00:23.770570 18372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "a99f26c0884c4246b7d51de4f925bb66"
format_stamp: "Formatted at 2026-08-12 03:00:23 on dist-test-slave-zpvx"
server_key: "83b975b82d97a6310f4ed0be7553ca56"
server_key_iv: "e32cdf9d79b5fd125939beb539616e2f"
server_key_version: "encryptionkey@0"
I20260812 03:00:23.770987 18372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:23.819401 18372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:23.825009 18372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:23.825485 18372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:23.828183 18372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:23.832343 18372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:23.832623 18372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 03:00:23.832894 18372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:23.833063 18372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:23.970037 18372 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:38111
I20260812 03:00:23.970165 18598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:38111 every 8 connection(s)
I20260812 03:00:23.972764 18372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:00:23.974959 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 18372
I20260812 03:00:23.975859 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:00:23.977484 10713 external_mini_cluster.cc:1428] Setting key a9935f9207bd8c1b2564fa945f79e07c
Aug 12 03:00:23 dist-test-slave-zpvx krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503623, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:24.009056 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:23.977591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:37497 (local address 127.10.118.126:38505)
0812 03:00:23.979362 (+ 1771us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:23.979395 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:23.981168 (+ 1773us) server_negotiation.cc:407] Connection header received
0812 03:00:23.986615 (+ 5447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:23.986649 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:23.986833 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:23.987015 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:23.989421 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:23.990367 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:23.992472 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:23.993158 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:23.993448 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:23.999378 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:23.999403 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:23.999419 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:23.999530 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:24.003268 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:24.004465 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:24.004487 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:24.004500 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:24.004706 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:24.005344 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:24.005362 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:24.005376 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:24.005867 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:24.006056 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:24.007147 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0812 03:00:24.007511 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":629,"threads_started":1}
I20260812 03:00:24.009766 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:23.977998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38505 (local address 127.10.118.67:37497)
0812 03:00:23.980938 (+ 2940us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:23.981022 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:23.986277 (+ 5255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:23.987293 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:23.987330 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:23.988150 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:23.989122 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:23.989177 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:23.990624 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:23.990645 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:23.992191 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:23.992230 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:23.993502 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:23.993539 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:23.994243 (+ 704us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:23.995369 (+ 1126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:23.995467 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:23.999060 (+ 3593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:24.003462 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:24.003488 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:24.003511 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:24.004202 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:24.004874 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:24.004893 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:24.004913 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:24.005173 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:24.006053 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:24.006083 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:24.006527 (+ 444us) client_negotiation.cc:769] Sending connection context
0812 03:00:24.006856 (+ 329us) client_negotiation.cc:240] Negotiation successful
0812 03:00:24.007351 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2329,"thread_start_us":1497,"threads_started":1}
I20260812 03:00:24.011708 18599 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38505
I20260812 03:00:24.012111 18599 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:24.013191 18599 heartbeater.cc:511] Master 127.10.118.126:38505 requested a full tablet report, sending...
I20260812 03:00:24.015303 17801 ts_manager.cc:194] Registered new tserver with Master: a99f26c0884c4246b7d51de4f925bb66 (127.10.118.67:38111)
I20260812 03:00:24.016804 17801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:37497
I20260812 03:00:24.026739 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 03:00:24 dist-test-slave-zpvx krb5kdc[17717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 03:00:24 dist-test-slave-zpvx krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 03:00:24.107170 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:24.080422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.10.118.65:46207)
0812 03:00:24.082596 (+ 2174us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:24.082648 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:24.082711 (+ 63us) server_negotiation.cc:407] Connection header received
0812 03:00:24.083061 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:24.083083 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:24.083348 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:24.083561 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:24.086425 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:24.087762 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:24.089882 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:24.090662 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:24.090949 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:24.096910 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:24.096939 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:24.096972 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:24.097078 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:24.101093 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:24.102211 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:24.102235 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:24.102250 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:24.102472 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:24.103151 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:24.103171 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:24.103184 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:24.103835 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:24.104314 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:24.105027 (+ 713us) server_negotiation.cc:299] Negotiation successful
0812 03:00:24.105534 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666,"thread_start_us":1270,"threads_started":1}
Aug 12 03:00:24 dist-test-slave-zpvx krb5kdc[17717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:24.136831 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:24.114585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.10.118.126:38505)
0812 03:00:24.115003 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:24.115018 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:24.115871 (+ 853us) server_negotiation.cc:407] Connection header received
0812 03:00:24.116188 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:24.116209 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:24.116477 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:24.116654 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:24.118562 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:24.119552 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:24.121335 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:24.121938 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:24.122160 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:24.127732 (+ 5572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:24.127757 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:24.127773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:24.127875 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:24.131652 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:24.132767 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:24.132789 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:24.132803 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:24.133006 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:24.133572 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:24.133592 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:24.133608 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:24.134131 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:24.134430 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:24.135049 (+ 619us) server_negotiation.cc:299] Negotiation successful
0812 03:00:24.135434 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260812 03:00:24.137917 17801 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314
I20260812 03:00:24.139647 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 17893
I20260812 03:00:24.174976 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 18138
I20260812 03:00:24.206100 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 18372
I20260812 03:00:24.237170 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 17738
2026-08-12T03:00:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (10060 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:00:24 dist-test-slave-zpvx krb5kdc[18620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:00:24 dist-test-slave-zpvx krb5kdc[18620](info): set up 2 sockets
Aug 12 03:00:24 dist-test-slave-zpvx krb5kdc[18620](info): commencing operation
krb5kdc: starting...
W20260812 03:00:25.633529 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.328s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:00:25 dist-test-slave-zpvx krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:00:25.869042 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38005
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:38005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:26.183633 18634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:26.184271 18634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:26.184929 18634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:26.215466 18634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:26.215760 18634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:26.215996 18634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:26.216197 18634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:26.216403 18634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:00:26.252919 18634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:38005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:26.254065 18634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:26.255106 18634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:26.272472 18642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:26.272486 18641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:26.272683 18644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:26.274847 18634 server_base.cc:1034] running on GCE node
I20260812 03:00:27.460211 18634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:27.462991 18634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:27.464478 18634 hybrid_clock.cc:648] HybridClock initialized: now 1786503627464444 us; error 68 us; skew 500 ppm
I20260812 03:00:27.465364 18634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:27 dist-test-slave-zpvx krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503627, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:27.471755 18634 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:00:27.475395 18634 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:27.475587 18634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:27.476001 18634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:27.481396 18634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "7dcd6f901ffa41d695cca81e85587706"
format_stamp: "Formatted at 2026-08-12 03:00:27 on dist-test-slave-zpvx"
server_key: "fbc37a413e066fd93120821a5299ad45"
server_key_iv: "7cc85745f25bdc6444c832482943e731"
server_key_version: "encryptionkey@0"
I20260812 03:00:27.482559 18634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "7dcd6f901ffa41d695cca81e85587706"
format_stamp: "Formatted at 2026-08-12 03:00:27 on dist-test-slave-zpvx"
server_key: "fbc37a413e066fd93120821a5299ad45"
server_key_iv: "7cc85745f25bdc6444c832482943e731"
server_key_version: "encryptionkey@0"
I20260812 03:00:27.489924 18634 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 03:00:27.495550 18653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:27.496707 18634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:00:27.497047 18634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "7dcd6f901ffa41d695cca81e85587706"
format_stamp: "Formatted at 2026-08-12 03:00:27 on dist-test-slave-zpvx"
server_key: "fbc37a413e066fd93120821a5299ad45"
server_key_iv: "7cc85745f25bdc6444c832482943e731"
server_key_version: "encryptionkey@0"
I20260812 03:00:27.497442 18634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:27.550635 18634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:27.556233 18634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:27.556731 18634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:27.626842 18634 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:38005
I20260812 03:00:27.626914 18736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:38005 every 8 connection(s)
I20260812 03:00:27.628067 18634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:27.633359 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:27.636421 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 18634
I20260812 03:00:27.636986 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:27.638567 10713 external_mini_cluster.cc:1428] Setting key d1e9506b142c45f31b0aa83078b3876f
I20260812 03:00:27.656119 18737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706: Bootstrap starting.
Aug 12 03:00:27 dist-test-slave-zpvx krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:27.663130 18737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:27.665294 18737 log.cc:824] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:27.670696 18737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706: No bootstrap required, opened a new log
I20260812 03:00:27.678182 18746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:27.641634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.10.118.126:38005)
0812 03:00:27.644498 (+ 2864us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:27.644526 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:27.644584 (+ 58us) server_negotiation.cc:407] Connection header received
0812 03:00:27.648556 (+ 3972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:27.648599 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:27.649014 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:27.649353 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:27.652642 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:27.654246 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:27.656880 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:27.657749 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:27.657991 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:27.665000 (+ 7009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:27.665023 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:27.665054 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:27.665152 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:27.670687 (+ 5535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:27.672098 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:27.672124 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:27.672140 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:27.672444 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:27.673173 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:27.673201 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:27.673216 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:27.674025 (+ 809us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:27.674355 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:27.675296 (+ 941us) server_negotiation.cc:299] Negotiation successful
0812 03:00:27.675784 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2379,"thread_start_us":1906,"threads_started":1}
I20260812 03:00:27.692207 18737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd6f901ffa41d695cca81e85587706" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38005 } }
I20260812 03:00:27.692981 18737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:27.693238 18737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcd6f901ffa41d695cca81e85587706, State: Initialized, Role: FOLLOWER
I20260812 03:00:27.694041 18737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd6f901ffa41d695cca81e85587706" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38005 } }
I20260812 03:00:27.694536 18737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:27.694761 18737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:27.695009 18737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:27.699007 18737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd6f901ffa41d695cca81e85587706" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38005 } }
I20260812 03:00:27.699673 18737 leader_election.cc:302] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dcd6f901ffa41d695cca81e85587706; no voters:
I20260812 03:00:27.701362 18737 leader_election.cc:288] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:27.701812 18754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:27.704411 18754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [term 1 LEADER]: Becoming Leader. State: Replica: 7dcd6f901ffa41d695cca81e85587706, State: Running, Role: LEADER
I20260812 03:00:27.705173 18754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd6f901ffa41d695cca81e85587706" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38005 } }
I20260812 03:00:27.705972 18737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:27.717063 18755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dcd6f901ffa41d695cca81e85587706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd6f901ffa41d695cca81e85587706" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38005 } } }
I20260812 03:00:27.718001 18755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:27.718545 18757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dcd6f901ffa41d695cca81e85587706. Latest consensus state: current_term: 1 leader_uuid: "7dcd6f901ffa41d695cca81e85587706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dcd6f901ffa41d695cca81e85587706" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38005 } } }
I20260812 03:00:27.719225 18757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:27.721419 18771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:27.732125 18771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:27.748854 18771 catalog_manager.cc:1347] Generated new cluster ID: 2c0589dc8734489abe17c5d6b24a8a53
I20260812 03:00:27.749222 18771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:27.775738 18771 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:27.777590 18771 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:27.793473 18771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706: Generated new TSK 0
I20260812 03:00:27.794375 18771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:00:27.857134 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38005
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:28.195621 18789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:28.196209 18789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:28.196957 18789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:28.229475 18789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:28.229890 18789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:28.231007 18789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:00:28.268682 18789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:28.270015 18789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:28.271190 18789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:28.292744 18801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:28.293294 18803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:28.294060 18807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:28.295722 18789 server_base.cc:1034] running on GCE node
I20260812 03:00:29.481392 18789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:29.483680 18789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:29.485074 18789 hybrid_clock.cc:648] HybridClock initialized: now 1786503629485058 us; error 63 us; skew 500 ppm
I20260812 03:00:29.485903 18789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:29 dist-test-slave-zpvx krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:29.499171 18789 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:00:29.502767 18789 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:29.503036 18789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:29.503512 18789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:29.509132 18789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf"
format_stamp: "Formatted at 2026-08-12 03:00:29 on dist-test-slave-zpvx"
server_key: "1360c7bfe63de5f3c7b14695b4d965be"
server_key_iv: "efb37b3a82dad67f58fe1e0c9fe92f7a"
server_key_version: "encryptionkey@0"
I20260812 03:00:29.510293 18789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf"
format_stamp: "Formatted at 2026-08-12 03:00:29 on dist-test-slave-zpvx"
server_key: "1360c7bfe63de5f3c7b14695b4d965be"
server_key_iv: "efb37b3a82dad67f58fe1e0c9fe92f7a"
server_key_version: "encryptionkey@0"
I20260812 03:00:29.518344 18789 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 03:00:29.523908 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:29.525133 18789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:00:29.525460 18789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf"
format_stamp: "Formatted at 2026-08-12 03:00:29 on dist-test-slave-zpvx"
server_key: "1360c7bfe63de5f3c7b14695b4d965be"
server_key_iv: "efb37b3a82dad67f58fe1e0c9fe92f7a"
server_key_version: "encryptionkey@0"
I20260812 03:00:29.525854 18789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:29.572858 18789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:29.578522 18789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:29.578987 18789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:29.581511 18789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:29.585503 18789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:29.585743 18789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:29.586004 18789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:29.586180 18789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:29.724022 18789 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:37027
I20260812 03:00:29.724156 19021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:37027 every 8 connection(s)
I20260812 03:00:29.725234 18789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:00:29.732926 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 18789
I20260812 03:00:29.733378 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:00:29.734516 10713 external_mini_cluster.cc:1428] Setting key 394aed95cc17cfd9ed9b6cbf9ef34f94
Aug 12 03:00:29 dist-test-slave-zpvx krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503629, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:00:29.760848 19025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:29.728913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:51655 (local address 127.10.118.126:38005)
0812 03:00:29.730421 (+ 1508us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:29.730449 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:29.732249 (+ 1800us) server_negotiation.cc:407] Connection header received
0812 03:00:29.737512 (+ 5263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:29.737539 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:29.737771 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:29.737962 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:29.740409 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:29.741333 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:29.743450 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:29.744033 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:29.744321 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:29.750002 (+ 5681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:29.750028 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:29.750044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:29.750142 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:29.753870 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:29.755109 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:29.755131 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:29.755148 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:29.755342 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:29.756059 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:29.756078 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:29.756091 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:29.756673 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:29.756925 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:29.758003 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0812 03:00:29.758493 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":561,"threads_started":1}
I20260812 03:00:29.761772 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:29.729202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38005 (local address 127.10.118.65:51655)
0812 03:00:29.731895 (+ 2693us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:29.731965 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:29.737158 (+ 5193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:29.738259 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:29.738297 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:29.739176 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:29.740088 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:29.740134 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:29.741603 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:29.741624 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:29.743181 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:29.743219 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:29.744490 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:29.744532 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:29.745170 (+ 638us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:29.746412 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:29.746484 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:29.749652 (+ 3168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:29.754112 (+ 4460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:29.754146 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:29.754171 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:29.754888 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:29.755532 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:29.755563 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:29.755583 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:29.755850 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:29.756889 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:29.756929 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:29.757401 (+ 472us) client_negotiation.cc:769] Sending connection context
0812 03:00:29.757781 (+ 380us) client_negotiation.cc:240] Negotiation successful
0812 03:00:29.758511 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2200,"thread_start_us":1512,"threads_started":1}
I20260812 03:00:29.763985 19022 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38005
I20260812 03:00:29.764454 19022 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:29.765364 19022 heartbeater.cc:511] Master 127.10.118.126:38005 requested a full tablet report, sending...
I20260812 03:00:29.768251 18683 ts_manager.cc:194] Registered new tserver with Master: 3df6ce8c35e34ead8aec4d8c6195a4bf (127.10.118.65:37027)
I20260812 03:00:29.771056 18683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:51655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:00:29.789469 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38005
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:30.094858 19031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:30.095413 19031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:30.096110 19031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:30.128217 19031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:30.128654 19031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:30.129575 19031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:00:30.166363 19031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:30.167501 19031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:30.168529 19031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:30.188596 19047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:30.775067 19022 heartbeater.cc:503] Master 127.10.118.126:38005 was elected leader, sending a full tablet report...
W20260812 03:00:30.188701 19042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:30.188792 19044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:31.358074 19046 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:00:31.358235 19031 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:00:31.362489 19031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:31.365321 19031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:31.366797 19031 hybrid_clock.cc:648] HybridClock initialized: now 1786503631366739 us; error 84 us; skew 500 ppm
I20260812 03:00:31.367596 19031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:31 dist-test-slave-zpvx krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503631, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:31.373677 19031 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:00:31.377213 19031 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:31.377451 19031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:31.377964 19031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:31.383385 19031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "afb7da65ac8d469ba859199b0439bc61"
format_stamp: "Formatted at 2026-08-12 03:00:31 on dist-test-slave-zpvx"
server_key: "7f5c0ebbf9f2eb47fcdb07e92f182eb4"
server_key_iv: "e4e73e45c23d67b23f2932e7ebd44093"
server_key_version: "encryptionkey@0"
I20260812 03:00:31.384617 19031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "afb7da65ac8d469ba859199b0439bc61"
format_stamp: "Formatted at 2026-08-12 03:00:31 on dist-test-slave-zpvx"
server_key: "7f5c0ebbf9f2eb47fcdb07e92f182eb4"
server_key_iv: "e4e73e45c23d67b23f2932e7ebd44093"
server_key_version: "encryptionkey@0"
I20260812 03:00:31.391716 19031 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260812 03:00:31.397137 19059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:31.398128 19031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 03:00:31.398461 19031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "afb7da65ac8d469ba859199b0439bc61"
format_stamp: "Formatted at 2026-08-12 03:00:31 on dist-test-slave-zpvx"
server_key: "7f5c0ebbf9f2eb47fcdb07e92f182eb4"
server_key_iv: "e4e73e45c23d67b23f2932e7ebd44093"
server_key_version: "encryptionkey@0"
I20260812 03:00:31.398838 19031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:31.432596 19031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:31.438215 19031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:31.438695 19031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:31.441227 19031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:31.445429 19031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:31.445652 19031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:31.445947 19031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:31.446094 19031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:31.585453 19031 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:36931
I20260812 03:00:31.585551 19258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:36931 every 8 connection(s)
I20260812 03:00:31.586635 19031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:00:31.594879 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 19031
I20260812 03:00:31.595461 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:00:31.596925 10713 external_mini_cluster.cc:1428] Setting key 55762491d3d8c16dd6f12dc30532049e
Aug 12 03:00:31 dist-test-slave-zpvx krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503631, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:00:31.628755 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:31.590280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:55373 (local address 127.10.118.126:38005)
0812 03:00:31.591796 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:31.591810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:31.594020 (+ 2210us) server_negotiation.cc:407] Connection header received
0812 03:00:31.600633 (+ 6613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:31.600664 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:31.600893 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:31.601075 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:31.604484 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:31.606103 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:31.609672 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:31.610778 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:31.611103 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:31.617000 (+ 5897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:31.617024 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:31.617043 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:31.617140 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:31.621502 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:31.622713 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:31.622749 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:31.622764 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:31.622964 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:31.623696 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:31.623716 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:31.623784 (+ 68us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:31.624493 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:31.624722 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:31.625931 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0812 03:00:31.626398 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":877,"threads_started":1}
I20260812 03:00:31.628957 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:31.590579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38005 (local address 127.10.118.66:55373)
0812 03:00:31.593775 (+ 3196us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:31.593846 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:31.600140 (+ 6294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:31.601427 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:31.601466 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:31.602673 (+ 1207us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:31.604035 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:31.604115 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:31.606486 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:31.606525 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:31.609337 (+ 2812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:31.609399 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:31.611170 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:31.611210 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:31.611871 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:31.613039 (+ 1168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:31.613111 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:31.616660 (+ 3549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:31.621717 (+ 5057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:31.621748 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:31.621785 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:31.622495 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:31.623169 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:31.623199 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:31.623216 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:31.623480 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:31.624735 (+ 1255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:31.624768 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:31.625240 (+ 472us) client_negotiation.cc:769] Sending connection context
0812 03:00:31.625652 (+ 412us) client_negotiation.cc:240] Negotiation successful
0812 03:00:31.626186 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2611,"thread_start_us":1755,"threads_started":1}
I20260812 03:00:31.631454 19259 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38005
I20260812 03:00:31.631955 19259 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:31.633111 19259 heartbeater.cc:511] Master 127.10.118.126:38005 requested a full tablet report, sending...
I20260812 03:00:31.635852 18683 ts_manager.cc:194] Registered new tserver with Master: afb7da65ac8d469ba859199b0439bc61 (127.10.118.66:36931)
I20260812 03:00:31.638010 18683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:55373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:00:31.660655 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38005
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:31.979511 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:31.980118 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:31.980852 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:32.013963 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:32.014400 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:32.015403 19270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:00:32.051221 19270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:32.052516 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:32.053609 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:32.072170 19280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:32.642773 19259 heartbeater.cc:503] Master 127.10.118.126:38005 was elected leader, sending a full tablet report...
W20260812 03:00:32.072202 19279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:32.073726 19270 server_base.cc:1034] running on GCE node
W20260812 03:00:32.072162 19282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:33.258246 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:33.260556 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:33.261937 19270 hybrid_clock.cc:648] HybridClock initialized: now 1786503633261906 us; error 52 us; skew 500 ppm
I20260812 03:00:33.262840 19270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:33 dist-test-slave-zpvx krb5kdc[18620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503633, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:33.269104 19270 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:00:33.272742 19270 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:33.272984 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:33.273451 19270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:33.278777 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c"
format_stamp: "Formatted at 2026-08-12 03:00:33 on dist-test-slave-zpvx"
server_key: "e44b1751be63a84f918758cd49614ac2"
server_key_iv: "4f7b34a4f83db348899009c897725ce4"
server_key_version: "encryptionkey@0"
I20260812 03:00:33.279951 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c"
format_stamp: "Formatted at 2026-08-12 03:00:33 on dist-test-slave-zpvx"
server_key: "e44b1751be63a84f918758cd49614ac2"
server_key_iv: "4f7b34a4f83db348899009c897725ce4"
server_key_version: "encryptionkey@0"
I20260812 03:00:33.287292 19270 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 03:00:33.293030 19295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:33.294023 19270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:00:33.294366 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c"
format_stamp: "Formatted at 2026-08-12 03:00:33 on dist-test-slave-zpvx"
server_key: "e44b1751be63a84f918758cd49614ac2"
server_key_iv: "4f7b34a4f83db348899009c897725ce4"
server_key_version: "encryptionkey@0"
I20260812 03:00:33.294811 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:33.337031 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:33.342989 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:33.343479 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:33.346268 19270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:33.350482 19270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:33.350723 19270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:33.350988 19270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:33.351177 19270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:33.489141 19270 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:38393
I20260812 03:00:33.489250 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:38393 every 8 connection(s)
I20260812 03:00:33.490321 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:00:33.499408 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 19270
I20260812 03:00:33.500190 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:00:33.501740 10713 external_mini_cluster.cc:1428] Setting key ce613d7b94498265bbad72e7634b60e8
Aug 12 03:00:33 dist-test-slave-zpvx krb5kdc[18620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503633, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:33.526508 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.494115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:59819 (local address 127.10.118.126:38005)
0812 03:00:33.495128 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.495143 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.497102 (+ 1959us) server_negotiation.cc:407] Connection header received
0812 03:00:33.503741 (+ 6639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.503779 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.503968 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.504150 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:33.507175 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.508287 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.510298 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.510933 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.511129 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.515861 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:33.515894 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:33.515911 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:33.516011 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:33.520755 (+ 4744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:33.522216 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:33.522239 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:33.522252 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:33.522455 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:33.523079 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:33.523097 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:33.523110 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:33.523589 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:33.523765 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.524759 (+ 994us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.525094 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":441,"threads_started":1}
I20260812 03:00:33.527019 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.494481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38005 (local address 127.10.118.67:59819)
0812 03:00:33.496875 (+ 2394us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:33.496945 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:33.503370 (+ 6425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:33.504438 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:33.504475 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:33.505733 (+ 1258us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:33.506882 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.506927 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.508546 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.508566 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.510081 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.510118 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.511204 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.511228 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.511555 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.512360 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:00:33.512490 (+ 130us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:33.515515 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:33.521013 (+ 5498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:33.521048 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:33.521083 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:33.521973 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:33.522619 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:00:33.522637 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:00:33.522651 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:00:33.522912 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:00:33.523765 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:33.523793 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:33.524184 (+ 391us) client_negotiation.cc:769] Sending connection context
0812 03:00:33.524566 (+ 382us) client_negotiation.cc:240] Negotiation successful
0812 03:00:33.525031 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1908,"thread_start_us":1132,"threads_started":1}
I20260812 03:00:33.528726 19503 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38005
I20260812 03:00:33.529131 19503 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:33.529968 19503 heartbeater.cc:511] Master 127.10.118.126:38005 requested a full tablet report, sending...
I20260812 03:00:33.532048 18683 ts_manager.cc:194] Registered new tserver with Master: 5a9ade7b62bb4d56a4bb25c8ab43ff7c (127.10.118.67:38393)
I20260812 03:00:33.533978 18683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:59819
I20260812 03:00:33.535573 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:00:33.571689 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.551915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.10.118.126:38005)
0812 03:00:33.552556 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.552589 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.553470 (+ 881us) server_negotiation.cc:407] Connection header received
0812 03:00:33.553806 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.553824 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.553986 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.554116 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:33.556407 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.557304 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.559108 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.559679 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.559868 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.563297 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:33.563318 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:33.563334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:33.563431 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:33.566699 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:33.567695 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:33.567715 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:33.567728 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:33.567920 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:33.568552 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:33.568570 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:33.568586 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:33.569095 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:33.569383 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.569939 (+ 556us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.570248 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 03:00:33.577890 18681 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:00:33.598879 18681 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:00:33.665130 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.628041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37027 (local address 127.0.0.1:48226)
0812 03:00:33.642963 (+ 14922us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:33.643241 (+ 278us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:33.643481 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:33.648724 (+ 5243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:33.648751 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:33.648843 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:00:33.649683 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.649737 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.654378 (+ 4641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.654422 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.659068 (+ 4646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.659124 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.661942 (+ 2818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.661981 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.662441 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.662617 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 03:00:33.662827 (+ 210us) client_negotiation.cc:240] Negotiation successful
0812 03:00:33.663171 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14478,"mutex_wait_us":1043,"thread_start_us":15541,"threads_started":1}
I20260812 03:00:33.665755 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.626175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.10.118.67:38393)
0812 03:00:33.643060 (+ 16885us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.643090 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.643145 (+ 55us) server_negotiation.cc:407] Connection header received
0812 03:00:33.643347 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.643387 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.643767 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.643964 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:00:33.652209 (+ 8245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.654583 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.660362 (+ 5779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.661683 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.662018 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.662483 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.663659 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.663968 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16379,"thread_start_us":15953,"threads_started":1}
I20260812 03:00:33.666189 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.630008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.10.118.66:36931)
0812 03:00:33.637599 (+ 7591us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.637641 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.650304 (+ 12663us) server_negotiation.cc:407] Connection header received
0812 03:00:33.650795 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.650820 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.651052 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.651270 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:00:33.652532 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.653991 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.660472 (+ 6481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.662451 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.662776 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.663103 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.664263 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.664600 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6984,"thread_start_us":6556,"threads_started":1}
I20260812 03:00:33.671169 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.629922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.10.118.65:37027)
0812 03:00:33.647212 (+ 17290us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.647255 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.647327 (+ 72us) server_negotiation.cc:407] Connection header received
0812 03:00:33.647569 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.647612 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.648052 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.648310 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:00:33.650626 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.652705 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.659433 (+ 6728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.661570 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.661895 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.662384 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.668583 (+ 6199us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.669018 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16609,"thread_start_us":18794,"threads_started":1}
I20260812 03:00:33.672107 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.625892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:38393 (local address 127.0.0.1:38684)
0812 03:00:33.636303 (+ 10411us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:33.639469 (+ 3166us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:33.639864 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:33.644077 (+ 4213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:33.644120 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:33.644265 (+ 145us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:00:33.645128 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.645206 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.654837 (+ 9631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.654876 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.660078 (+ 5202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.660127 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.662781 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.662815 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.663216 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.663368 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 03:00:33.663521 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 03:00:33.663857 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9752,"mutex_wait_us":1820,"thread_start_us":23385,"threads_started":1}
I20260812 03:00:33.673337 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.628332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:36931 (local address 127.0.0.1:46812)
0812 03:00:33.650090 (+ 21758us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:33.650134 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:33.650501 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:33.651292 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:33.651321 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:33.651419 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:00:33.652233 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.652282 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.654248 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.654285 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.659626 (+ 5341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.659677 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.663369 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.663391 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.663816 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.663961 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 03:00:33.666729 (+ 2768us) client_negotiation.cc:240] Negotiation successful
0812 03:00:33.667019 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21349,"mutex_wait_us":1732,"thread_start_us":6512,"threads_started":1}
I20260812 03:00:33.680696 19388 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf732e345d040cc8a10bb22f1e4441e (DEFAULT_TABLE table=test-table [id=5c2a1e2e46ad4e8b9da7ce78d3805374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:00:33.680699 18905 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf732e345d040cc8a10bb22f1e4441e (DEFAULT_TABLE table=test-table [id=5c2a1e2e46ad4e8b9da7ce78d3805374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:00:33.681269 19148 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf732e345d040cc8a10bb22f1e4441e (DEFAULT_TABLE table=test-table [id=5c2a1e2e46ad4e8b9da7ce78d3805374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:00:33.682678 19388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf732e345d040cc8a10bb22f1e4441e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:33.682675 18905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf732e345d040cc8a10bb22f1e4441e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:33.683192 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf732e345d040cc8a10bb22f1e4441e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:33.725749 19544 tablet_bootstrap.cc:492] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Bootstrap starting.
I20260812 03:00:33.721992 19543 tablet_bootstrap.cc:492] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf: Bootstrap starting.
I20260812 03:00:33.733817 19545 tablet_bootstrap.cc:492] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Bootstrap starting.
I20260812 03:00:33.737234 19544 tablet_bootstrap.cc:654] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:33.739408 19543 tablet_bootstrap.cc:654] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:33.739651 19544 log.cc:824] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:33.740247 19545 tablet_bootstrap.cc:654] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:33.741583 19543 log.cc:824] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:33.742861 19545 log.cc:824] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:33.752887 19545 tablet_bootstrap.cc:492] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: No bootstrap required, opened a new log
I20260812 03:00:33.753453 19545 ts_tablet_manager.cc:1397] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260812 03:00:33.753666 19544 tablet_bootstrap.cc:492] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: No bootstrap required, opened a new log
I20260812 03:00:33.754174 19544 ts_tablet_manager.cc:1397] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.000s
I20260812 03:00:33.755425 19543 tablet_bootstrap.cc:492] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf: No bootstrap required, opened a new log
I20260812 03:00:33.755968 19543 ts_tablet_manager.cc:1397] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.006s
I20260812 03:00:33.778632 19543 raft_consensus.cc:348] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.779369 19543 raft_consensus.cc:374] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:33.779668 19543 raft_consensus.cc:729] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3df6ce8c35e34ead8aec4d8c6195a4bf, State: Initialized, Role: FOLLOWER
I20260812 03:00:33.781011 19543 consensus_queue.cc:260] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.784420 19545 raft_consensus.cc:348] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.784415 19544 raft_consensus.cc:348] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.785187 19543 ts_tablet_manager.cc:1428] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf: Time spent starting tablet: real 0.029s user 0.020s sys 0.008s
I20260812 03:00:33.785308 19544 raft_consensus.cc:374] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:33.785396 19545 raft_consensus.cc:374] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:33.785665 19544 raft_consensus.cc:729] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9ade7b62bb4d56a4bb25c8ab43ff7c, State: Initialized, Role: FOLLOWER
I20260812 03:00:33.785800 19545 raft_consensus.cc:729] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb7da65ac8d469ba859199b0439bc61, State: Initialized, Role: FOLLOWER
I20260812 03:00:33.786576 19544 consensus_queue.cc:260] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.786764 19545 consensus_queue.cc:260] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.790876 19544 ts_tablet_manager.cc:1428] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Time spent starting tablet: real 0.036s user 0.027s sys 0.008s
I20260812 03:00:33.791735 19503 heartbeater.cc:503] Master 127.10.118.126:38005 was elected leader, sending a full tablet report...
I20260812 03:00:33.791980 19545 ts_tablet_manager.cc:1428] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Time spent starting tablet: real 0.038s user 0.033s sys 0.004s
W20260812 03:00:33.843086 19260 tablet.cc:2369] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:00:33.861831 19550 raft_consensus.cc:482] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:00:33.862282 19550 raft_consensus.cc:504] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.864984 19550 leader_election.cc:288] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9ade7b62bb4d56a4bb25c8ab43ff7c (127.10.118.67:38393), afb7da65ac8d469ba859199b0439bc61 (127.10.118.66:36931)
I20260812 03:00:33.884298 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.865515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:52989 (local address 127.10.118.67:38393)
0812 03:00:33.866142 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.866179 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.868984 (+ 2805us) server_negotiation.cc:407] Connection header received
0812 03:00:33.869548 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.869574 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.869833 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.870002 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:00:33.871717 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.873063 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.878685 (+ 5622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.880318 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.880714 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.880994 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.882606 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.882885 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260812 03:00:33.884307 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.866025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:58399 (local address 127.10.118.66:36931)
0812 03:00:33.866695 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.866728 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.869082 (+ 2354us) server_negotiation.cc:407] Connection header received
0812 03:00:33.869457 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.869484 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.869753 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.869930 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:00:33.871634 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.873062 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.878686 (+ 5624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.880328 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.880724 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.880994 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.882608 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.882878 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260812 03:00:33.885507 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.865390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:38393 (local address 127.10.118.65:52989)
0812 03:00:33.868777 (+ 3387us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:33.868840 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:33.869272 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:33.870052 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:33.870086 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:33.870220 (+ 134us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:00:33.871422 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.871476 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.873383 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.873419 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.878345 (+ 4926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.878403 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.880731 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.880768 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.882039 (+ 1271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.882363 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 03:00:33.882528 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 03:00:33.883120 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2846,"mutex_wait_us":1815,"spinlock_wait_cycles":150656,"thread_start_us":2323,"threads_started":1}
I20260812 03:00:33.885507 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.865564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:36931 (local address 127.10.118.65:58399)
0812 03:00:33.868788 (+ 3224us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:33.868844 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:33.869174 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:33.870021 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:33.870054 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:33.870223 (+ 169us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:00:33.871290 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.871350 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.873383 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.873416 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.878386 (+ 4970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:33.878431 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.880755 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.880787 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:33.882057 (+ 1270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.882387 (+ 330us) client_negotiation.cc:769] Sending connection context
0812 03:00:33.882527 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 03:00:33.883132 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2718,"mutex_wait_us":2114,"spinlock_wait_cycles":19072,"thread_start_us":2409,"threads_started":1}
I20260812 03:00:33.887367 19420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf732e345d040cc8a10bb22f1e4441e" candidate_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" is_pre_election: true
I20260812 03:00:33.887775 19181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf732e345d040cc8a10bb22f1e4441e" candidate_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb7da65ac8d469ba859199b0439bc61" is_pre_election: true
I20260812 03:00:33.888207 19420 raft_consensus.cc:2454] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df6ce8c35e34ead8aec4d8c6195a4bf in term 0.
I20260812 03:00:33.888599 19181 raft_consensus.cc:2454] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3df6ce8c35e34ead8aec4d8c6195a4bf in term 0.
I20260812 03:00:33.889488 18824 leader_election.cc:302] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df6ce8c35e34ead8aec4d8c6195a4bf, 5a9ade7b62bb4d56a4bb25c8ab43ff7c; no voters:
I20260812 03:00:33.890245 19550 raft_consensus.cc:2781] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:00:33.890533 19550 raft_consensus.cc:482] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:00:33.890810 19550 raft_consensus.cc:3037] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:33.895391 19550 raft_consensus.cc:504] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.896894 19550 leader_election.cc:288] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [CANDIDATE]: Term 1 election: Requested vote from peers 5a9ade7b62bb4d56a4bb25c8ab43ff7c (127.10.118.67:38393), afb7da65ac8d469ba859199b0439bc61 (127.10.118.66:36931)
I20260812 03:00:33.897639 19420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf732e345d040cc8a10bb22f1e4441e" candidate_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c"
I20260812 03:00:33.897836 19181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf732e345d040cc8a10bb22f1e4441e" candidate_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb7da65ac8d469ba859199b0439bc61"
I20260812 03:00:33.898083 19420 raft_consensus.cc:3037] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:33.898312 19181 raft_consensus.cc:3037] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:33.902963 19181 raft_consensus.cc:2454] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df6ce8c35e34ead8aec4d8c6195a4bf in term 1.
I20260812 03:00:33.903398 19420 raft_consensus.cc:2454] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3df6ce8c35e34ead8aec4d8c6195a4bf in term 1.
I20260812 03:00:33.903960 18826 leader_election.cc:302] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3df6ce8c35e34ead8aec4d8c6195a4bf, afb7da65ac8d469ba859199b0439bc61; no voters:
I20260812 03:00:33.904811 19550 raft_consensus.cc:2781] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:33.906499 19550 raft_consensus.cc:686] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [term 1 LEADER]: Becoming Leader. State: Replica: 3df6ce8c35e34ead8aec4d8c6195a4bf, State: Running, Role: LEADER
I20260812 03:00:33.907439 19550 consensus_queue.cc:237] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } }
I20260812 03:00:33.918769 18681 catalog_manager.cc:5495] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3df6ce8c35e34ead8aec4d8c6195a4bf (127.10.118.65). New cstate: current_term: 1 leader_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3df6ce8c35e34ead8aec4d8c6195a4bf" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37027 } health_report { overall_health: HEALTHY } } }
I20260812 03:00:33.964193 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:33.956273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.10.118.65:37027)
0812 03:00:33.956683 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:33.956703 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:33.956744 (+ 41us) server_negotiation.cc:407] Connection header received
0812 03:00:33.957099 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:33.957117 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:33.957360 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:33.957506 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:00:33.958127 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.958978 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.960687 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:33.961237 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:33.961457 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:33.962157 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:00:33.962647 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:00:33.962784 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:33.963072 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 03:00:33.963223 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 03:00:33.976415 19181 raft_consensus.cc:1261] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 1 FOLLOWER]: Refusing update from remote peer 3df6ce8c35e34ead8aec4d8c6195a4bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:00:33.976899 19420 raft_consensus.cc:1261] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 1 FOLLOWER]: Refusing update from remote peer 3df6ce8c35e34ead8aec4d8c6195a4bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:00:33.977988 19555 consensus_queue.cc:1035] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb7da65ac8d469ba859199b0439bc61" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:00:33.978632 19550 consensus_queue.cc:1035] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9ade7b62bb4d56a4bb25c8ab43ff7c" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 03:00:33.983577 19024 tablet.cc:2369] T 3cf732e345d040cc8a10bb22f1e4441e P 3df6ce8c35e34ead8aec4d8c6195a4bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 03:00:33.995525 19505 tablet.cc:2369] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:00:34.006335 19564 mvcc.cc:205] Tried to move back new op lower bound from 7317518884752773120 to 7317518884492824576. Current Snapshot: MvccSnapshot[applied={T|T < 7317518884752773120}]
I20260812 03:00:34.032171 18681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:00:34.032776 18681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:00:34.041292 18681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dcd6f901ffa41d695cca81e85587706: Sending DeleteTablet for 3 replicas of tablet 3cf732e345d040cc8a10bb22f1e4441e
I20260812 03:00:34.043162 19388 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf732e345d040cc8a10bb22f1e4441e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:00:34 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:38684
I20260812 03:00:34.043393 19148 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf732e345d040cc8a10bb22f1e4441e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:00:34 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:46812
I20260812 03:00:34.043690 18905 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cf732e345d040cc8a10bb22f1e4441e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:00:34 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:48226
I20260812 03:00:34.053670 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 18789
I20260812 03:00:34.063361 19580 tablet_replica.cc:330] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: stopping tablet replica
I20260812 03:00:34.064183 19580 raft_consensus.cc:2229] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:00:34.064869 19580 raft_consensus.cc:2258] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:00:34.068820 19580 ts_tablet_manager.cc:1905] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:00:34.070575 19581 tablet_replica.cc:330] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: stopping tablet replica
I20260812 03:00:34.071281 19581 raft_consensus.cc:2229] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:00:34.071866 19581 raft_consensus.cc:2258] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:00:34.074980 19581 ts_tablet_manager.cc:1905] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:00:34.084796 19581 ts_tablet_manager.cc:1918] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:00:34.085134 19581 log.cc:1196] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/3cf732e345d040cc8a10bb22f1e4441e
I20260812 03:00:34.085752 19581 ts_tablet_manager.cc:1939] T 3cf732e345d040cc8a10bb22f1e4441e P afb7da65ac8d469ba859199b0439bc61: Deleting consensus metadata
I20260812 03:00:34.086490 19580 ts_tablet_manager.cc:1918] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:00:34.086921 19580 log.cc:1196] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/3cf732e345d040cc8a10bb22f1e4441e
I20260812 03:00:34.087733 19580 ts_tablet_manager.cc:1939] T 3cf732e345d040cc8a10bb22f1e4441e P 5a9ade7b62bb4d56a4bb25c8ab43ff7c: Deleting consensus metadata
I20260812 03:00:34.088502 18657 catalog_manager.cc:4841] TS afb7da65ac8d469ba859199b0439bc61 (127.10.118.66:36931): tablet 3cf732e345d040cc8a10bb22f1e4441e (table test-table [id=5c2a1e2e46ad4e8b9da7ce78d3805374]) successfully deleted
I20260812 03:00:34.090948 18654 catalog_manager.cc:4841] TS 5a9ade7b62bb4d56a4bb25c8ab43ff7c (127.10.118.67:38393): tablet 3cf732e345d040cc8a10bb22f1e4441e (table test-table [id=5c2a1e2e46ad4e8b9da7ce78d3805374]) successfully deleted
W20260812 03:00:34.106312 18655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:37027 (error 108)
I20260812 03:00:34.106546 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 19031
I20260812 03:00:34.123723 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:34.122620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37027 (local address 127.0.0.1:48256)
0812 03:00:34.123121 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:34.123304 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260812 03:00:34.124133 18655 catalog_manager.cc:4584] TS 3df6ce8c35e34ead8aec4d8c6195a4bf (127.10.118.65:37027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cf732e345d040cc8a10bb22f1e4441e: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37027: connect: Connection refused (error 111)
I20260812 03:00:34.144793 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 19270
I20260812 03:00:34.179592 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:34.178476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37027 (local address 127.0.0.1:48270)
0812 03:00:34.178968 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:34.179142 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260812 03:00:34.182291 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:34.181323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37027 (local address 127.0.0.1:48282)
0812 03:00:34.181774 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:34.181917 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 03:00:34.184827 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 18634
2026-08-12T03:00:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (9946 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T03:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:34Z Disabled control of system clock
I20260812 03:00:34.282591 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36605
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:36605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 03:00:34.591925 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:34.592597 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:34.593189 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:34.623549 19594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:34.623930 19594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:34.624253 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:34.624673 19594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:34.624943 19594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:00:34.659482 19594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:36605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:34.660722 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:34.662410 19594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:34.680893 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:34.681077 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:34.682569 19594 server_base.cc:1034] running on GCE node
W20260812 03:00:34.682489 19608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:35.875777 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:35.878477 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:35.879887 19594 hybrid_clock.cc:648] HybridClock initialized: now 1786503635879841 us; error 52 us; skew 500 ppm
I20260812 03:00:35.880797 19594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:00:35.893088 19594 webserver.cc:460] Webserver started at http://127.10.118.126:34361/ using document root <none> and password file <none>
I20260812 03:00:35.894063 19594 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:35.894302 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:35.894863 19594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:35.900259 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "de6bc5c75a574addb18f60793cc8d4f8"
format_stamp: "Formatted at 2026-08-12 03:00:35 on dist-test-slave-zpvx"
server_key: "a660f27d4f4e49375ef26a4c032bbc1a"
server_key_iv: "3b6e5f69b850da255f447772ea473c43"
server_key_version: "encryptionkey@0"
I20260812 03:00:35.901491 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "de6bc5c75a574addb18f60793cc8d4f8"
format_stamp: "Formatted at 2026-08-12 03:00:35 on dist-test-slave-zpvx"
server_key: "a660f27d4f4e49375ef26a4c032bbc1a"
server_key_iv: "3b6e5f69b850da255f447772ea473c43"
server_key_version: "encryptionkey@0"
I20260812 03:00:35.908802 19594 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:00:35.914479 19623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:35.915529 19594 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:00:35.915860 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "de6bc5c75a574addb18f60793cc8d4f8"
format_stamp: "Formatted at 2026-08-12 03:00:35 on dist-test-slave-zpvx"
server_key: "a660f27d4f4e49375ef26a4c032bbc1a"
server_key_iv: "3b6e5f69b850da255f447772ea473c43"
server_key_version: "encryptionkey@0"
I20260812 03:00:35.916244 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:35.942660 19594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:35.944160 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:35.944662 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:36.021318 19594 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:36605
I20260812 03:00:36.021422 19703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:36605 every 8 connection(s)
I20260812 03:00:36.024049 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:36.029369 19704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:36.029933 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 19594
I20260812 03:00:36.030402 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:36.031522 10713 external_mini_cluster.cc:1428] Setting key 8c4ad8576564631d74d8406629019630
I20260812 03:00:36.047904 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:36.034087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.10.118.126:36605)
0812 03:00:36.036785 (+ 2698us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:36.036808 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:36.036867 (+ 59us) server_negotiation.cc:407] Connection header received
0812 03:00:36.042245 (+ 5378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:36.042300 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:36.042765 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:36.043223 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:36.044777 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:36.044809 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:36.044849 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:36.044911 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:36.045030 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:36.045090 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:36.045388 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:36.045540 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 03:00:36.045820 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1991,"thread_start_us":2121,"threads_started":1}
I20260812 03:00:36.055640 19704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8: Bootstrap starting.
I20260812 03:00:36.061149 19704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:36.063139 19704 log.cc:824] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:36.068018 19704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8: No bootstrap required, opened a new log
I20260812 03:00:36.088280 19704 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bc5c75a574addb18f60793cc8d4f8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36605 } }
I20260812 03:00:36.089023 19704 raft_consensus.cc:374] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:36.089327 19704 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6bc5c75a574addb18f60793cc8d4f8, State: Initialized, Role: FOLLOWER
I20260812 03:00:36.090027 19704 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bc5c75a574addb18f60793cc8d4f8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36605 } }
I20260812 03:00:36.090575 19704 raft_consensus.cc:388] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:36.090917 19704 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:36.091243 19704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:36.095489 19704 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bc5c75a574addb18f60793cc8d4f8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36605 } }
I20260812 03:00:36.096163 19704 leader_election.cc:302] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de6bc5c75a574addb18f60793cc8d4f8; no voters:
I20260812 03:00:36.098218 19704 leader_election.cc:288] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:36.099166 19711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:36.102905 19711 raft_consensus.cc:686] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [term 1 LEADER]: Becoming Leader. State: Replica: de6bc5c75a574addb18f60793cc8d4f8, State: Running, Role: LEADER
I20260812 03:00:36.103721 19711 consensus_queue.cc:237] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bc5c75a574addb18f60793cc8d4f8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36605 } }
I20260812 03:00:36.104112 19704 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:36.115085 19713 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6bc5c75a574addb18f60793cc8d4f8. Latest consensus state: current_term: 1 leader_uuid: "de6bc5c75a574addb18f60793cc8d4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bc5c75a574addb18f60793cc8d4f8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36605 } } }
I20260812 03:00:36.115873 19713 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:36.118088 19712 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de6bc5c75a574addb18f60793cc8d4f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6bc5c75a574addb18f60793cc8d4f8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36605 } } }
I20260812 03:00:36.118558 19712 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:36.119771 19727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:36.131330 19727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:36.148180 19727 catalog_manager.cc:1347] Generated new cluster ID: 16bc296f61aa443196e3741540e9a117
I20260812 03:00:36.148476 19727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:36.174978 19727 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:36.175611 19727 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:36.189815 19727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8: Generated new TSK 0
I20260812 03:00:36.190689 19727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:00:36.199705 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:36605
--builtin_ntp_servers=127.10.118.84:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 03:00:36.515548 19746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:36.516068 19746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:36.516709 19746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:36.547730 19746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:36.548022 19746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:36.551065 19746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:00:36.585247 19746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:36605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:36.586480 19746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:36.588063 19746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:36.608775 19767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:36.613464 19746 server_base.cc:1034] running on GCE node
W20260812 03:00:36.612649 19761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:36.609108 19773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:37.816282 19746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:37.819306 19746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:37.820729 19746 hybrid_clock.cc:648] HybridClock initialized: now 1786503637820700 us; error 55 us; skew 500 ppm
I20260812 03:00:37.821558 19746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:00:37.828917 19746 webserver.cc:460] Webserver started at http://127.10.118.65:37797/ using document root <none> and password file <none>
I20260812 03:00:37.829885 19746 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:37.830097 19746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:37.830595 19746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:37.835898 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "0cab661e8a42464b90fc1431315deade"
format_stamp: "Formatted at 2026-08-12 03:00:37 on dist-test-slave-zpvx"
server_key: "cffe0f9f27f0d1148b83414f69147986"
server_key_iv: "036c10a3700c7c74e66fe004fa008abc"
server_key_version: "encryptionkey@0"
I20260812 03:00:37.837121 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "0cab661e8a42464b90fc1431315deade"
format_stamp: "Formatted at 2026-08-12 03:00:37 on dist-test-slave-zpvx"
server_key: "cffe0f9f27f0d1148b83414f69147986"
server_key_iv: "036c10a3700c7c74e66fe004fa008abc"
server_key_version: "encryptionkey@0"
I20260812 03:00:37.845075 19746 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:00:37.851081 19788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:37.852216 19746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 03:00:37.852656 19746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "0cab661e8a42464b90fc1431315deade"
format_stamp: "Formatted at 2026-08-12 03:00:37 on dist-test-slave-zpvx"
server_key: "cffe0f9f27f0d1148b83414f69147986"
server_key_iv: "036c10a3700c7c74e66fe004fa008abc"
server_key_version: "encryptionkey@0"
I20260812 03:00:37.853215 19746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:37.879279 19746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:37.880897 19746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:37.881364 19746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:37.883955 19746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:37.888085 19746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:37.888322 19746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:37.888666 19746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:37.888855 19746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:38.036751 19746 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:38349
I20260812 03:00:38.036886 19961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:38349 every 8 connection(s)
I20260812 03:00:38.040843 19746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:00:38.047019 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 19746
I20260812 03:00:38.047690 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:00:38.049669 10713 external_mini_cluster.cc:1428] Setting key e5d425b50ddafb3ea1a96b65433e53ac
I20260812 03:00:38.057696 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:36605
--builtin_ntp_servers=127.10.118.84:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 03:00:38.075119 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:38.049975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:60863 (local address 127.10.118.126:36605)
0812 03:00:38.062590 (+ 12615us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:38.062618 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:38.063335 (+ 717us) server_negotiation.cc:407] Connection header received
0812 03:00:38.069282 (+ 5947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:38.069319 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:38.069509 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:38.069706 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:38.072026 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:38.072062 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:38.072097 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:38.072182 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:38.072291 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:38.072352 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:38.072634 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:38.073075 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 03:00:38.073234 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12151,"thread_start_us":12544,"threads_started":1}
I20260812 03:00:38.076047 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:38.048134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:36605 (local address 127.10.118.65:60863)
0812 03:00:38.063105 (+ 14971us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:38.063173 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:38.068774 (+ 5601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:38.070042 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:38.070091 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:38.070313 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:38.071374 (+ 1061us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:38.071405 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:38.071506 (+ 101us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:38.071708 (+ 202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:38.072595 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:38.072655 (+ 60us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:38.072694 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 03:00:38.073143 (+ 449us) client_negotiation.cc:240] Negotiation successful
0812 03:00:38.073426 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14278,"thread_start_us":16057,"threads_started":1}
I20260812 03:00:38.078649 19962 heartbeater.cc:347] Connected to a master server at 127.10.118.126:36605
I20260812 03:00:38.079203 19962 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:38.080322 19962 heartbeater.cc:511] Master 127.10.118.126:36605 requested a full tablet report, sending...
I20260812 03:00:38.083442 19652 ts_manager.cc:194] Registered new tserver with Master: 0cab661e8a42464b90fc1431315deade (127.10.118.65:38349)
W20260812 03:00:38.383837 19970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:38.384450 19970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:38.385075 19970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:38.415463 19970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:38.415751 19970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:38.416677 19970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:00:38.451690 19970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:36605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:38.453009 19970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:38.454620 19970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:38.472443 19987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:39.087143 19962 heartbeater.cc:503] Master 127.10.118.126:36605 was elected leader, sending a full tablet report...
W20260812 03:00:38.472455 19989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:38.474551 19970 server_base.cc:1034] running on GCE node
W20260812 03:00:38.473084 19984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:39.654567 19970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:39.657423 19970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:39.658874 19970 hybrid_clock.cc:648] HybridClock initialized: now 1786503639658812 us; error 75 us; skew 500 ppm
I20260812 03:00:39.659667 19970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:00:39.666978 19970 webserver.cc:460] Webserver started at http://127.10.118.66:40883/ using document root <none> and password file <none>
I20260812 03:00:39.667873 19970 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:39.668116 19970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:39.668684 19970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:39.674315 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "46dacb2f6ba749c8a21941a2c4a6d820"
format_stamp: "Formatted at 2026-08-12 03:00:39 on dist-test-slave-zpvx"
server_key: "401381d493cbd94a5911e5a409179627"
server_key_iv: "100b4fd0886aaefa7692b4ff0efea46a"
server_key_version: "encryptionkey@0"
I20260812 03:00:39.675498 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "46dacb2f6ba749c8a21941a2c4a6d820"
format_stamp: "Formatted at 2026-08-12 03:00:39 on dist-test-slave-zpvx"
server_key: "401381d493cbd94a5911e5a409179627"
server_key_iv: "100b4fd0886aaefa7692b4ff0efea46a"
server_key_version: "encryptionkey@0"
I20260812 03:00:39.682711 19970 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 03:00:39.688287 20001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:39.689388 19970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 03:00:39.689718 19970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "46dacb2f6ba749c8a21941a2c4a6d820"
format_stamp: "Formatted at 2026-08-12 03:00:39 on dist-test-slave-zpvx"
server_key: "401381d493cbd94a5911e5a409179627"
server_key_iv: "100b4fd0886aaefa7692b4ff0efea46a"
server_key_version: "encryptionkey@0"
I20260812 03:00:39.690107 19970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:39.715854 19970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:39.717373 19970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:39.717877 19970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:39.720536 19970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:39.724617 19970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:39.724843 19970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:39.725100 19970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:39.725289 19970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:39.863286 19970 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:42763
I20260812 03:00:39.863401 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:42763 every 8 connection(s)
I20260812 03:00:39.865954 19970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:00:39.868103 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 19970
I20260812 03:00:39.868690 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:00:39.870198 10713 external_mini_cluster.cc:1428] Setting key 6a39abfeb9e1f360733bcf8e233dbc0d
I20260812 03:00:39.876261 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:36605
--builtin_ntp_servers=127.10.118.84:41721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 03:00:39.884331 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:39.870837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:44309 (local address 127.10.118.126:36605)
0812 03:00:39.872471 (+ 1634us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:39.872502 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:39.874943 (+ 2441us) server_negotiation.cc:407] Connection header received
0812 03:00:39.879642 (+ 4699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:39.879676 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:39.879860 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:39.880054 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:39.881785 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:39.881823 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:39.881863 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:39.881941 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:39.882048 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:39.882109 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:39.882359 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:39.882563 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 03:00:39.882704 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":843,"threads_started":1}
I20260812 03:00:39.884320 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:39.871259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:36605 (local address 127.10.118.66:44309)
0812 03:00:39.874724 (+ 3465us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:39.874792 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:39.879276 (+ 4484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:39.880259 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:39.880283 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:39.880463 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:39.881311 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:39.881327 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:39.881388 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:39.881503 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:39.882295 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:39.882325 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:39.882344 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 03:00:39.882636 (+ 292us) client_negotiation.cc:240] Negotiation successful
0812 03:00:39.882810 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2722,"thread_start_us":1859,"threads_started":1}
I20260812 03:00:39.886251 20212 heartbeater.cc:347] Connected to a master server at 127.10.118.126:36605
I20260812 03:00:39.886648 20212 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:39.887398 20212 heartbeater.cc:511] Master 127.10.118.126:36605 requested a full tablet report, sending...
I20260812 03:00:39.889622 19652 ts_manager.cc:194] Registered new tserver with Master: 46dacb2f6ba749c8a21941a2c4a6d820 (127.10.118.66:42763)
W20260812 03:00:40.191001 20225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:40.191537 20225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:40.192144 20225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:40.222939 20225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:40.223224 20225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:40.224146 20225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:00:40.258338 20225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:36605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:40.259569 20225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:40.261224 20225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:40.279211 20232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:40.892468 20212 heartbeater.cc:503] Master 127.10.118.126:36605 was elected leader, sending a full tablet report...
W20260812 03:00:40.279987 20231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:40.282040 20225 server_base.cc:1034] running on GCE node
W20260812 03:00:40.280107 20234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:41.477118 20225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:41.479979 20225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:41.481426 20225 hybrid_clock.cc:648] HybridClock initialized: now 1786503641481384 us; error 59 us; skew 500 ppm
I20260812 03:00:41.482201 20225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:00:41.493984 20225 webserver.cc:460] Webserver started at http://127.10.118.67:39119/ using document root <none> and password file <none>
I20260812 03:00:41.494895 20225 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:41.495077 20225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:41.495499 20225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:41.500845 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "2bfad3d28caa44148af5c14ffb2bfa03"
format_stamp: "Formatted at 2026-08-12 03:00:41 on dist-test-slave-zpvx"
server_key: "dfc8f5bbf2b79f26dd39cd78b2381231"
server_key_iv: "bff1a7fa9c0de043ad038f3cee6d6250"
server_key_version: "encryptionkey@0"
I20260812 03:00:41.501962 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "2bfad3d28caa44148af5c14ffb2bfa03"
format_stamp: "Formatted at 2026-08-12 03:00:41 on dist-test-slave-zpvx"
server_key: "dfc8f5bbf2b79f26dd39cd78b2381231"
server_key_iv: "bff1a7fa9c0de043ad038f3cee6d6250"
server_key_version: "encryptionkey@0"
I20260812 03:00:41.509560 20225 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 03:00:41.515126 20247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:41.516181 20225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 03:00:41.516543 20225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "2bfad3d28caa44148af5c14ffb2bfa03"
format_stamp: "Formatted at 2026-08-12 03:00:41 on dist-test-slave-zpvx"
server_key: "dfc8f5bbf2b79f26dd39cd78b2381231"
server_key_iv: "bff1a7fa9c0de043ad038f3cee6d6250"
server_key_version: "encryptionkey@0"
I20260812 03:00:41.516909 20225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:41.542199 20225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:41.543749 20225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:41.544224 20225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:41.546764 20225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:00:41.550884 20225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:00:41.551111 20225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:41.551390 20225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:00:41.551568 20225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:41.690003 20225 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:45571
I20260812 03:00:41.690105 20462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:45571 every 8 connection(s)
I20260812 03:00:41.692690 20225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:00:41.696190 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 20225
I20260812 03:00:41.696769 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:00:41.698264 10713 external_mini_cluster.cc:1428] Setting key f5e2df91d89db50cf713e7529812381b
I20260812 03:00:41.710912 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.697309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:49117 (local address 127.10.118.126:36605)
0812 03:00:41.698825 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:41.698862 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:41.700810 (+ 1948us) server_negotiation.cc:407] Connection header received
0812 03:00:41.706728 (+ 5918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:41.706771 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:41.706945 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:41.707114 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:41.708758 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:41.708783 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:41.708804 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:41.708848 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:41.708929 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:41.708968 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:41.709261 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:41.709497 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 03:00:41.709606 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":547,"threads_started":1}
I20260812 03:00:41.711190 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.697480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:36605 (local address 127.10.118.67:49117)
0812 03:00:41.700577 (+ 3097us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:41.700661 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:41.706352 (+ 5691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:41.707328 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:41.707352 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:41.707467 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:41.708273 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:41.708290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:41.708347 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:41.708523 (+ 176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:41.709234 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:41.709266 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:41.709291 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 03:00:41.709533 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 03:00:41.709693 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2370,"thread_start_us":1552,"threads_started":1}
I20260812 03:00:41.713060 20463 heartbeater.cc:347] Connected to a master server at 127.10.118.126:36605
I20260812 03:00:41.713450 20463 heartbeater.cc:464] Registering TS with master...
I20260812 03:00:41.714289 20463 heartbeater.cc:511] Master 127.10.118.126:36605 requested a full tablet report, sending...
I20260812 03:00:41.716900 19652 ts_manager.cc:194] Registered new tserver with Master: 2bfad3d28caa44148af5c14ffb2bfa03 (127.10.118.67:45571)
I20260812 03:00:41.718468 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:00:41.735718 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.730314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.10.118.126:36605)
0812 03:00:41.730735 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:41.730759 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:41.731679 (+ 920us) server_negotiation.cc:407] Connection header received
0812 03:00:41.732085 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:41.732105 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:41.732244 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:41.732432 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:41.733831 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:41.733851 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:41.733867 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:41.733909 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:41.733979 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:41.734011 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:41.734248 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:41.734484 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 03:00:41.734619 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 03:00:41.742138 19652 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:00:41.763108 19652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:00:41.803670 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.789815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:45571 (local address 127.0.0.1:42686)
0812 03:00:41.795089 (+ 5274us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:41.795191 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:41.795494 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:41.798894 (+ 3400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:41.798937 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:41.799074 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:41.799943 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:41.799979 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:41.800090 (+ 111us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:41.800279 (+ 189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:41.801504 (+ 1225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:41.801575 (+ 71us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:41.801612 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 03:00:41.801872 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 03:00:41.801974 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4642,"thread_start_us":1412,"threads_started":1}
I20260812 03:00:41.803964 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.789966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.10.118.67:45571)
0812 03:00:41.792729 (+ 2763us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:41.792768 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:41.795713 (+ 2945us) server_negotiation.cc:407] Connection header received
0812 03:00:41.795875 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:41.795904 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:41.796216 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:41.799304 (+ 3088us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:41.800990 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:41.801023 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:41.801061 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:41.801129 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:41.801253 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:41.801315 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:41.801595 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:41.801912 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 03:00:41.802084 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2237,"thread_start_us":1831,"threads_started":1}
I20260812 03:00:41.806597 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.791269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:42763 (local address 127.0.0.1:49662)
0812 03:00:41.795631 (+ 4362us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:41.795882 (+ 251us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:41.796101 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:41.798550 (+ 2449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:41.798584 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:41.798737 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:41.801680 (+ 2943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:41.801696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:41.801756 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:41.801883 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:41.802904 (+ 1021us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:41.802934 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:41.802963 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 03:00:41.804618 (+ 1655us) client_negotiation.cc:240] Negotiation successful
0812 03:00:41.804740 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3896,"thread_start_us":2821,"threads_started":1}
I20260812 03:00:41.807783 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.792041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.10.118.66:42763)
0812 03:00:41.794421 (+ 2380us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:41.794454 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:41.796313 (+ 1859us) server_negotiation.cc:407] Connection header received
0812 03:00:41.796522 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:41.796553 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:41.796812 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:41.797014 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:41.802176 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:41.802208 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:41.802247 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:41.802308 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:41.802427 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:41.802488 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:41.802770 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:41.803251 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 03:00:41.803408 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1896,"thread_start_us":1513,"threads_started":1}
I20260812 03:00:41.808962 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.796440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:38349 (local address 127.0.0.1:36868)
0812 03:00:41.798686 (+ 2246us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:41.801140 (+ 2454us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:41.801345 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:41.804551 (+ 3206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:41.804582 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:41.804679 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:41.805532 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:41.805557 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:41.805628 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:41.805741 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:41.806842 (+ 1101us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:41.806871 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:41.806908 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 03:00:41.807135 (+ 227us) client_negotiation.cc:240] Negotiation successful
0812 03:00:41.807260 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1819,"thread_start_us":1552,"threads_started":1}
I20260812 03:00:41.808995 20353 tablet_service.cc:1467] Processing CreateTablet for tablet d986ca45ad09429ba4564338e6a70ffb (DEFAULT_TABLE table=test-table [id=7a3ba29b112041d092ba9424b6f6c929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:00:41.810276 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.796829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.10.118.65:38349)
0812 03:00:41.802194 (+ 5365us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:41.802237 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:41.802287 (+ 50us) server_negotiation.cc:407] Connection header received
0812 03:00:41.802446 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:41.802473 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:41.802734 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:41.802933 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:41.806130 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:41.806162 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:41.806198 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:41.806270 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:41.806396 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:41.806457 (+ 61us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:41.806721 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:41.808436 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0812 03:00:41.808609 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4183,"thread_start_us":3854,"threads_started":1}
I20260812 03:00:41.811743 20353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d986ca45ad09429ba4564338e6a70ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:41.812286 20091 tablet_service.cc:1467] Processing CreateTablet for tablet d986ca45ad09429ba4564338e6a70ffb (DEFAULT_TABLE table=test-table [id=7a3ba29b112041d092ba9424b6f6c929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:00:41.814491 19857 tablet_service.cc:1467] Processing CreateTablet for tablet d986ca45ad09429ba4564338e6a70ffb (DEFAULT_TABLE table=test-table [id=7a3ba29b112041d092ba9424b6f6c929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:00:41.815361 20091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d986ca45ad09429ba4564338e6a70ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:41.815922 19857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d986ca45ad09429ba4564338e6a70ffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:41.837306 20495 tablet_bootstrap.cc:492] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade: Bootstrap starting.
I20260812 03:00:41.842224 20496 tablet_bootstrap.cc:492] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Bootstrap starting.
I20260812 03:00:41.844645 20495 tablet_bootstrap.cc:654] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:41.847451 20497 tablet_bootstrap.cc:492] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Bootstrap starting.
I20260812 03:00:41.848260 20495 log.cc:824] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:41.849334 20496 tablet_bootstrap.cc:654] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:41.851663 20496 log.cc:824] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:41.855751 20497 tablet_bootstrap.cc:654] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:41.856523 20495 tablet_bootstrap.cc:492] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade: No bootstrap required, opened a new log
I20260812 03:00:41.856940 20495 ts_tablet_manager.cc:1397] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20260812 03:00:41.857863 20496 tablet_bootstrap.cc:492] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: No bootstrap required, opened a new log
I20260812 03:00:41.858006 20497 log.cc:824] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:41.858342 20496 ts_tablet_manager.cc:1397] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20260812 03:00:41.877808 20497 tablet_bootstrap.cc:492] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: No bootstrap required, opened a new log
I20260812 03:00:41.878376 20497 ts_tablet_manager.cc:1397] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.012s
I20260812 03:00:41.893318 20496 raft_consensus.cc:348] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:41.894272 20496 raft_consensus.cc:374] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:41.894589 20496 raft_consensus.cc:729] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bfad3d28caa44148af5c14ffb2bfa03, State: Initialized, Role: FOLLOWER
I20260812 03:00:41.895486 20496 consensus_queue.cc:260] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:41.895697 20495 raft_consensus.cc:348] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:41.896595 20495 raft_consensus.cc:374] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:41.896942 20495 raft_consensus.cc:729] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cab661e8a42464b90fc1431315deade, State: Initialized, Role: FOLLOWER
I20260812 03:00:41.897922 20495 consensus_queue.cc:260] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:41.909632 20463 heartbeater.cc:503] Master 127.10.118.126:36605 was elected leader, sending a full tablet report...
I20260812 03:00:41.910498 20496 ts_tablet_manager.cc:1428] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Time spent starting tablet: real 0.052s user 0.037s sys 0.014s
I20260812 03:00:41.915748 20497 raft_consensus.cc:348] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:41.916683 20497 raft_consensus.cc:374] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:41.916982 20497 raft_consensus.cc:729] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46dacb2f6ba749c8a21941a2c4a6d820, State: Initialized, Role: FOLLOWER
I20260812 03:00:41.917871 20497 consensus_queue.cc:260] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:41.920235 20495 ts_tablet_manager.cc:1428] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade: Time spent starting tablet: real 0.063s user 0.040s sys 0.009s
I20260812 03:00:41.922973 20497 ts_tablet_manager.cc:1428] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Time spent starting tablet: real 0.044s user 0.034s sys 0.008s
W20260812 03:00:41.947306 20466 tablet.cc:2369] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:00:41.993678 20504 raft_consensus.cc:482] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:00:41.994225 20504 raft_consensus.cc:504] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:41.996718 20504 leader_election.cc:288] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bfad3d28caa44148af5c14ffb2bfa03 (127.10.118.67:45571), 46dacb2f6ba749c8a21941a2c4a6d820 (127.10.118.66:42763)
I20260812 03:00:42.005172 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.997397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:36423 (local address 127.10.118.67:45571)
0812 03:00:41.997956 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:41.997986 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:42.000393 (+ 2407us) server_negotiation.cc:407] Connection header received
0812 03:00:42.000796 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:42.000824 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:42.001024 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:42.001209 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:42.002912 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:42.002932 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:42.002952 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:42.002992 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:42.003094 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:42.003143 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:42.003366 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:42.003515 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 03:00:42.003658 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 03:00:42.005709 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.997123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:45571 (local address 127.10.118.65:36423)
0812 03:00:42.000134 (+ 3011us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:42.000209 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:42.000500 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:42.001337 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:42.001366 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:42.001481 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:42.002420 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:42.002448 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:42.002514 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:42.002672 (+ 158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:42.003319 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:42.003339 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:42.003355 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 03:00:42.003570 (+ 215us) client_negotiation.cc:240] Negotiation successful
0812 03:00:42.003684 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2522,"thread_start_us":2688,"threads_started":1}
I20260812 03:00:42.006016 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.997348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:42763 (local address 127.10.118.65:32917)
0812 03:00:42.000802 (+ 3454us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:42.000852 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:42.001056 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:42.001831 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:42.001868 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:42.001983 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:00:42.002722 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:00:42.002747 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:00:42.002819 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:00:42.002928 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:00:42.003646 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:00:42.003674 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:00:42.003700 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 03:00:42.003933 (+ 233us) client_negotiation.cc:240] Negotiation successful
0812 03:00:42.004040 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2894,"thread_start_us":2056,"threads_started":1}
I20260812 03:00:42.006075 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:41.997684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:32917 (local address 127.10.118.66:42763)
0812 03:00:41.998307 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:41.998344 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:42.001000 (+ 2656us) server_negotiation.cc:407] Connection header received
0812 03:00:42.001286 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:42.001315 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:42.001519 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:42.001703 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:42.003160 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:42.003201 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:42.003240 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:42.003309 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:42.003393 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:42.003451 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:42.003668 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:42.003909 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 03:00:42.004039 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260812 03:00:42.007786 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d986ca45ad09429ba4564338e6a70ffb" candidate_uuid: "0cab661e8a42464b90fc1431315deade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" is_pre_election: true
I20260812 03:00:42.007776 20125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d986ca45ad09429ba4564338e6a70ffb" candidate_uuid: "0cab661e8a42464b90fc1431315deade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" is_pre_election: true
I20260812 03:00:42.008746 20388 raft_consensus.cc:2454] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cab661e8a42464b90fc1431315deade in term 0.
I20260812 03:00:42.008771 20125 raft_consensus.cc:2454] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cab661e8a42464b90fc1431315deade in term 0.
I20260812 03:00:42.010135 19793 leader_election.cc:302] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cab661e8a42464b90fc1431315deade, 46dacb2f6ba749c8a21941a2c4a6d820; no voters:
I20260812 03:00:42.010893 20504 raft_consensus.cc:2781] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:00:42.011181 20504 raft_consensus.cc:482] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:00:42.011435 20504 raft_consensus.cc:3037] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:42.015928 20504 raft_consensus.cc:504] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:42.017305 20504 leader_election.cc:288] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [CANDIDATE]: Term 1 election: Requested vote from peers 2bfad3d28caa44148af5c14ffb2bfa03 (127.10.118.67:45571), 46dacb2f6ba749c8a21941a2c4a6d820 (127.10.118.66:42763)
I20260812 03:00:42.018239 20388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d986ca45ad09429ba4564338e6a70ffb" candidate_uuid: "0cab661e8a42464b90fc1431315deade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bfad3d28caa44148af5c14ffb2bfa03"
I20260812 03:00:42.018303 20125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d986ca45ad09429ba4564338e6a70ffb" candidate_uuid: "0cab661e8a42464b90fc1431315deade" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46dacb2f6ba749c8a21941a2c4a6d820"
I20260812 03:00:42.018716 20388 raft_consensus.cc:3037] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:42.018784 20125 raft_consensus.cc:3037] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:42.023459 20125 raft_consensus.cc:2454] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cab661e8a42464b90fc1431315deade in term 1.
I20260812 03:00:42.024276 19793 leader_election.cc:302] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cab661e8a42464b90fc1431315deade, 46dacb2f6ba749c8a21941a2c4a6d820; no voters:
I20260812 03:00:42.025040 20504 raft_consensus.cc:2781] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:42.025315 20388 raft_consensus.cc:2454] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cab661e8a42464b90fc1431315deade in term 1.
I20260812 03:00:42.027480 20504 raft_consensus.cc:686] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [term 1 LEADER]: Becoming Leader. State: Replica: 0cab661e8a42464b90fc1431315deade, State: Running, Role: LEADER
I20260812 03:00:42.028299 20504 consensus_queue.cc:237] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } }
I20260812 03:00:42.042824 19649 catalog_manager.cc:5495] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cab661e8a42464b90fc1431315deade (127.10.118.65). New cstate: current_term: 1 leader_uuid: "0cab661e8a42464b90fc1431315deade" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cab661e8a42464b90fc1431315deade" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38349 } health_report { overall_health: HEALTHY } } }
W20260812 03:00:42.057832 19965 tablet.cc:2369] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:00:42.124043 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:42.118837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.10.118.65:38349)
0812 03:00:42.119222 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:42.119240 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:42.119331 (+ 91us) server_negotiation.cc:407] Connection header received
0812 03:00:42.119791 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:42.119836 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:42.120123 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:42.120329 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:42.121929 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:42.121964 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:42.121996 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:42.122072 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:42.122179 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:42.122256 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:42.122521 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:42.122796 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 03:00:42.122938 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260812 03:00:42.125133 20215 tablet.cc:2369] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:00:42.136569 20125 raft_consensus.cc:1261] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 1 FOLLOWER]: Refusing update from remote peer 0cab661e8a42464b90fc1431315deade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:00:42.137148 20388 raft_consensus.cc:1261] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 1 FOLLOWER]: Refusing update from remote peer 0cab661e8a42464b90fc1431315deade: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:00:42.138309 20511 consensus_queue.cc:1035] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dacb2f6ba749c8a21941a2c4a6d820" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:00:42.138883 20504 consensus_queue.cc:1035] T d986ca45ad09429ba4564338e6a70ffb P 0cab661e8a42464b90fc1431315deade [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bfad3d28caa44148af5c14ffb2bfa03" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 45571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:00:42.158157 20515 mvcc.cc:205] Tried to move back new op lower bound from 7317518918176710656 to 7317518917756342272. Current Snapshot: MvccSnapshot[applied={T|T < 7317518918176710656}]
I20260812 03:00:42.192102 19651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:00:42.192706 19651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:00:42.202111 19651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de6bc5c75a574addb18f60793cc8d4f8: Sending DeleteTablet for 3 replicas of tablet d986ca45ad09429ba4564338e6a70ffb
I20260812 03:00:42.204025 20353 tablet_service.cc:1514] Processing DeleteTablet for tablet d986ca45ad09429ba4564338e6a70ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:00:42 UTC) from {username='slave'} at 127.0.0.1:42686
I20260812 03:00:42.204314 20091 tablet_service.cc:1514] Processing DeleteTablet for tablet d986ca45ad09429ba4564338e6a70ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:00:42 UTC) from {username='slave'} at 127.0.0.1:49662
I20260812 03:00:42.206781 19857 tablet_service.cc:1514] Processing DeleteTablet for tablet d986ca45ad09429ba4564338e6a70ffb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:00:42 UTC) from {username='slave'} at 127.0.0.1:36868
I20260812 03:00:42.207998 20535 tablet_replica.cc:330] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: stopping tablet replica
I20260812 03:00:42.208885 20535 raft_consensus.cc:2229] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:00:42.209795 20535 raft_consensus.cc:2258] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:00:42.212689 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 19746
I20260812 03:00:42.223388 20536 tablet_replica.cc:330] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: stopping tablet replica
I20260812 03:00:42.223381 20535 ts_tablet_manager.cc:1905] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:00:42.224189 20536 raft_consensus.cc:2229] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:00:42.224927 20536 raft_consensus.cc:2258] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:00:42.228242 20536 ts_tablet_manager.cc:1905] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:00:42.238456 20535 ts_tablet_manager.cc:1918] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:00:42.238870 20535 log.cc:1196] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/d986ca45ad09429ba4564338e6a70ffb
I20260812 03:00:42.239670 20535 ts_tablet_manager.cc:1939] T d986ca45ad09429ba4564338e6a70ffb P 2bfad3d28caa44148af5c14ffb2bfa03: Deleting consensus metadata
I20260812 03:00:42.242892 19629 catalog_manager.cc:4841] TS 2bfad3d28caa44148af5c14ffb2bfa03 (127.10.118.67:45571): tablet d986ca45ad09429ba4564338e6a70ffb (table test-table [id=7a3ba29b112041d092ba9424b6f6c929]) successfully deleted
I20260812 03:00:42.243528 20536 ts_tablet_manager.cc:1918] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:00:42.243924 20536 log.cc:1196] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/d986ca45ad09429ba4564338e6a70ffb
I20260812 03:00:42.244719 20536 ts_tablet_manager.cc:1939] T d986ca45ad09429ba4564338e6a70ffb P 46dacb2f6ba749c8a21941a2c4a6d820: Deleting consensus metadata
I20260812 03:00:42.247215 19631 catalog_manager.cc:4841] TS 46dacb2f6ba749c8a21941a2c4a6d820 (127.10.118.66:42763): tablet d986ca45ad09429ba4564338e6a70ffb (table test-table [id=7a3ba29b112041d092ba9424b6f6c929]) successfully deleted
I20260812 03:00:42.253640 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 19970
W20260812 03:00:42.253419 19626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:38349 (error 108)
I20260812 03:00:42.262693 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:42.261140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:38349 (local address 127.0.0.1:36896)
0812 03:00:42.261794 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:42.262064 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260812 03:00:42.263254 19626 catalog_manager.cc:4584] TS 0cab661e8a42464b90fc1431315deade (127.10.118.65:38349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d986ca45ad09429ba4564338e6a70ffb: Network error: Client connection negotiation failed: client connection to 127.10.118.65:38349: connect: Connection refused (error 111)
I20260812 03:00:42.287953 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 20225
I20260812 03:00:42.318763 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:42.317148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:38349 (local address 127.0.0.1:36902)
0812 03:00:42.317814 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:42.318054 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 03:00:42.321712 20488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:42.320246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:38349 (local address 127.0.0.1:36918)
0812 03:00:42.320863 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:42.321110 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 03:00:42.322422 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 19594
2026-08-12T03:00:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (8141 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:540: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestJwtMiniCluster (6 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 03:00:42.399995 10713 mini_oidc.cc:152] Starting JWKS server
I20260812 03:00:42.403779 10713 webserver.cc:460] Webserver started at https://127.0.0.1:40037/ using document root <none> and password file <none>
I20260812 03:00:42.404285 10713 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 03:00:42.405517 10713 webserver.cc:460] Webserver started at http://127.0.0.1:45749/ using document root <none> and password file <none>
2026-08-12T03:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:42Z Disabled control of system clock
I20260812 03:00:42.441006 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43959
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:42497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:43959
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40037/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/testchainca.cert with env {}
W20260812 03:00:42.753408 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:42.754050 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:42.754665 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:42.785611 20551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:42.785979 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:42.786206 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:42.786481 20551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:42.786753 20551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 03:00:42.787027 20551 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 03:00:42.787228 20551 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40037/jwks.json
I20260812 03:00:42.823578 20551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:42497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:43959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40037/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:42.824793 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:42.825841 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:42.844483 20564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:42.844491 20566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:42.846418 20551 server_base.cc:1034] running on GCE node
W20260812 03:00:42.844911 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:44.041023 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:44.044231 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:44.045710 20551 hybrid_clock.cc:648] HybridClock initialized: now 1786503644045676 us; error 46 us; skew 500 ppm
I20260812 03:00:44.046494 20551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:00:44.053423 20551 webserver.cc:460] Webserver started at http://127.10.118.126:35069/ using document root <none> and password file <none>
I20260812 03:00:44.054337 20551 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:44.054522 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:44.054936 20551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:44.060273 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "8440515c4efa4cd1aa8531dca4cadb45"
format_stamp: "Formatted at 2026-08-12 03:00:44 on dist-test-slave-zpvx"
server_key: "1eacdbc64856598c1ab93477464e7edd"
server_key_iv: "7aced2417b0c8c0b4d10cf2b77789ab8"
server_key_version: "encryptionkey@0"
I20260812 03:00:44.061408 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "8440515c4efa4cd1aa8531dca4cadb45"
format_stamp: "Formatted at 2026-08-12 03:00:44 on dist-test-slave-zpvx"
server_key: "1eacdbc64856598c1ab93477464e7edd"
server_key_iv: "7aced2417b0c8c0b4d10cf2b77789ab8"
server_key_version: "encryptionkey@0"
I20260812 03:00:44.068636 20551 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:00:44.074334 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:44.075399 20551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 03:00:44.075680 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "8440515c4efa4cd1aa8531dca4cadb45"
format_stamp: "Formatted at 2026-08-12 03:00:44 on dist-test-slave-zpvx"
server_key: "1eacdbc64856598c1ab93477464e7edd"
server_key_iv: "7aced2417b0c8c0b4d10cf2b77789ab8"
server_key_version: "encryptionkey@0"
I20260812 03:00:44.075994 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:44.119026 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:44.120594 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:44.121011 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:44.191942 20551 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:43959
I20260812 03:00:44.192051 20674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:43959 every 8 connection(s)
I20260812 03:00:44.194777 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:44.196874 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 20551
I20260812 03:00:44.197824 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:44.199348 10713 external_mini_cluster.cc:1428] Setting key 3486f1ec627c73a630931e5d6c6454f7
I20260812 03:00:44.201056 20675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:44.226779 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:44.201656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.10.118.126:43959)
0812 03:00:44.204441 (+ 2785us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:44.204479 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:44.204554 (+ 75us) server_negotiation.cc:407] Connection header received
0812 03:00:44.211762 (+ 7208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:44.211822 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:44.212394 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:44.212872 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:44.213999 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:44.216632 (+ 2633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:44.218573 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:44.219898 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:44.220257 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:44.221947 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:44.221986 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:44.222029 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:44.223204 (+ 1175us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:44.223338 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:44.223395 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:44.223558 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:44.223873 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 03:00:44.224254 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2052,"thread_start_us":1659,"threads_started":1}
I20260812 03:00:44.229353 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45: Bootstrap starting.
I20260812 03:00:44.231598 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:00:44.237311 20675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:44.239737 20675 log.cc:824] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:44.245812 20675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45: No bootstrap required, opened a new log
I20260812 03:00:44.266839 20675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8440515c4efa4cd1aa8531dca4cadb45" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43959 } }
I20260812 03:00:44.267562 20675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:44.267887 20675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8440515c4efa4cd1aa8531dca4cadb45, State: Initialized, Role: FOLLOWER
I20260812 03:00:44.268744 20675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8440515c4efa4cd1aa8531dca4cadb45" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43959 } }
I20260812 03:00:44.269313 20675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:44.269619 20675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:44.269941 20675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:44.274080 20675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8440515c4efa4cd1aa8531dca4cadb45" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43959 } }
I20260812 03:00:44.274796 20675 leader_election.cc:302] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8440515c4efa4cd1aa8531dca4cadb45; no voters:
I20260812 03:00:44.276645 20675 leader_election.cc:288] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:44.277127 20682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:44.279793 20682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [term 1 LEADER]: Becoming Leader. State: Replica: 8440515c4efa4cd1aa8531dca4cadb45, State: Running, Role: LEADER
I20260812 03:00:44.280555 20682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8440515c4efa4cd1aa8531dca4cadb45" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43959 } }
I20260812 03:00:44.281652 20675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:44.292532 20683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8440515c4efa4cd1aa8531dca4cadb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8440515c4efa4cd1aa8531dca4cadb45" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43959 } } }
I20260812 03:00:44.293141 20683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:44.294255 20684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8440515c4efa4cd1aa8531dca4cadb45. Latest consensus state: current_term: 1 leader_uuid: "8440515c4efa4cd1aa8531dca4cadb45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8440515c4efa4cd1aa8531dca4cadb45" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43959 } } }
I20260812 03:00:44.295112 20684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:44.297600 20698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:44.312130 20698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:44.328923 20698 catalog_manager.cc:1347] Generated new cluster ID: 7753e5b680874abea4396178e37a496a
I20260812 03:00:44.329241 20698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:44.346345 20698 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:44.348011 20698 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:44.364264 20698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8440515c4efa4cd1aa8531dca4cadb45: Generated new TSK 0
I20260812 03:00:44.365290 20698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 03:00:44.438880 20729 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 03:00:44.439239 20729 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 03:00:44.441984 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:44.402485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.10.118.126:43959)
0812 03:00:44.403165 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:44.403203 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:44.404943 (+ 1740us) server_negotiation.cc:407] Connection header received
0812 03:00:44.405504 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:44.405531 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:44.405726 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:44.405866 (+ 140us) server_negotiation.cc:226] Negotiated authn=JWT
0812 03:00:44.407169 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:44.408065 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:44.411485 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:44.412055 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:44.412290 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:44.413506 (+ 1216us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 03:00:44.439887 (+ 26381us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 03:00:44.440511 (+ 624us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59170: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":138}
W20260812 03:00:44.442270 20680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59170: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 03:00:44.442848 20726 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:00:44.402266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43959 (local address 127.0.0.1:59170)
0812 03:00:44.404763 (+ 2497us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:44.404820 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:44.405207 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:44.405994 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:44.406033 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:44.406102 (+ 69us) client_negotiation.cc:189] Negotiated authn=JWT
0812 03:00:44.406857 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:44.406936 (+ 79us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:44.408301 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:44.408338 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:44.411182 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:44.411240 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:44.412438 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:44.412478 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:44.412931 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:44.413142 (+ 211us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 03:00:44.440303 (+ 27161us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 03:00:44.440752 (+ 449us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.118.126:43959: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40037/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":1922,"thread_start_us":1553,"threads_started":1}
I20260812 03:00:44.445559 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 20551
2026-08-12T03:00:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (3273 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 03:00:45.674135 10713 mini_oidc.cc:152] Starting JWKS server
I20260812 03:00:45.677397 10713 webserver.cc:460] Webserver started at https://127.0.0.1:33431/ using document root <none> and password file <none>
I20260812 03:00:45.677893 10713 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 03:00:45.679723 10713 webserver.cc:460] Webserver started at http://127.0.0.1:33139/ using document root <none> and password file <none>
2026-08-12T03:00:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:45Z Disabled control of system clock
I20260812 03:00:45.713009 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:40241
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:40241
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33431/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 03:00:46.015256 20737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:46.015812 20737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:46.016306 20737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:46.051834 20737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:46.052129 20737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:46.052289 20737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:46.052556 20737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:46.052757 20737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 03:00:46.052954 20737 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 03:00:46.053095 20737 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33431/jwks.json
I20260812 03:00:46.087023 20737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:40241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33431/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:46.088125 20737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:46.089736 20737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:46.108793 20759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:46.108860 20763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:46.109807 20756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:46.110715 20737 server_base.cc:1034] running on GCE node
I20260812 03:00:47.307054 20737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:47.310665 20737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:47.312155 20737 hybrid_clock.cc:648] HybridClock initialized: now 1786503647312074 us; error 83 us; skew 500 ppm
I20260812 03:00:47.313090 20737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:00:47.323807 20737 webserver.cc:460] Webserver started at http://127.10.118.126:34791/ using document root <none> and password file <none>
I20260812 03:00:47.324800 20737 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:47.324986 20737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:47.325425 20737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:47.330706 20737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "e47ff39003d94941bd3277d9d8cfac19"
format_stamp: "Formatted at 2026-08-12 03:00:47 on dist-test-slave-zpvx"
server_key: "463051fe3e94aade0afdc516ba022d36"
server_key_iv: "42a1302c14c3b42ae5b26b4c1a3298ff"
server_key_version: "encryptionkey@0"
I20260812 03:00:47.331802 20737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "e47ff39003d94941bd3277d9d8cfac19"
format_stamp: "Formatted at 2026-08-12 03:00:47 on dist-test-slave-zpvx"
server_key: "463051fe3e94aade0afdc516ba022d36"
server_key_iv: "42a1302c14c3b42ae5b26b4c1a3298ff"
server_key_version: "encryptionkey@0"
I20260812 03:00:47.338984 20737 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 03:00:47.344776 20776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:47.345906 20737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 03:00:47.346274 20737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "e47ff39003d94941bd3277d9d8cfac19"
format_stamp: "Formatted at 2026-08-12 03:00:47 on dist-test-slave-zpvx"
server_key: "463051fe3e94aade0afdc516ba022d36"
server_key_iv: "42a1302c14c3b42ae5b26b4c1a3298ff"
server_key_version: "encryptionkey@0"
I20260812 03:00:47.346666 20737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:47.382939 20737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:47.384395 20737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:47.384862 20737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:47.453564 20737 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:40241
I20260812 03:00:47.453677 20862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:40241 every 8 connection(s)
I20260812 03:00:47.456331 20737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:47.461531 20864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:47.463048 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 20737
I20260812 03:00:47.463616 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:47.465210 10713 external_mini_cluster.cc:1428] Setting key 6c1a7bd414be80f420d7ef3c9028071c
I20260812 03:00:47.488543 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19: Bootstrap starting.
I20260812 03:00:47.488941 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:47.467744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.10.118.126:40241)
0812 03:00:47.470673 (+ 2929us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:47.470699 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:47.470752 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:00:47.475341 (+ 4589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:47.475383 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:47.475745 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:47.476076 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:47.477559 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:47.479244 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:47.482181 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:47.483023 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:47.483328 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:47.485309 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:47.485332 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:47.485358 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:47.486107 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:47.486200 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:47.486245 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:47.486368 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:47.486782 (+ 414us) server_negotiation.cc:299] Negotiation successful
0812 03:00:47.487097 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2406,"thread_start_us":1925,"threads_started":1}
I20260812 03:00:47.493908 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:00:47.495541 20864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:47.497886 20864 log.cc:824] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:47.503317 20864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19: No bootstrap required, opened a new log
I20260812 03:00:47.524614 20864 raft_consensus.cc:348] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47ff39003d94941bd3277d9d8cfac19" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:00:47.525317 20864 raft_consensus.cc:374] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:47.525584 20864 raft_consensus.cc:729] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47ff39003d94941bd3277d9d8cfac19, State: Initialized, Role: FOLLOWER
I20260812 03:00:47.526340 20864 consensus_queue.cc:260] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47ff39003d94941bd3277d9d8cfac19" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:00:47.526890 20864 raft_consensus.cc:388] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:47.527166 20864 raft_consensus.cc:482] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:47.527484 20864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:47.531740 20864 raft_consensus.cc:504] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47ff39003d94941bd3277d9d8cfac19" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:00:47.532429 20864 leader_election.cc:302] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e47ff39003d94941bd3277d9d8cfac19; no voters:
I20260812 03:00:47.534089 20864 leader_election.cc:288] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:47.534471 20872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:47.536690 20872 raft_consensus.cc:686] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [term 1 LEADER]: Becoming Leader. State: Replica: e47ff39003d94941bd3277d9d8cfac19, State: Running, Role: LEADER
I20260812 03:00:47.537627 20872 consensus_queue.cc:237] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47ff39003d94941bd3277d9d8cfac19" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:00:47.538477 20864 sys_catalog.cc:564] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:47.549432 20874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e47ff39003d94941bd3277d9d8cfac19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47ff39003d94941bd3277d9d8cfac19" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } } }
I20260812 03:00:47.550312 20874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:47.551115 20877 sys_catalog.cc:455] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e47ff39003d94941bd3277d9d8cfac19. Latest consensus state: current_term: 1 leader_uuid: "e47ff39003d94941bd3277d9d8cfac19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e47ff39003d94941bd3277d9d8cfac19" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } } }
I20260812 03:00:47.551760 20877 sys_catalog.cc:458] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:47.553448 20885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:47.565689 20885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:47.582309 20885 catalog_manager.cc:1347] Generated new cluster ID: 76b2ff3fb02a486e86841ba91eee7dca
I20260812 03:00:47.582629 20885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:47.615803 20885 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:47.618408 20885 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:47.641034 20885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e47ff39003d94941bd3277d9d8cfac19: Generated new TSK 0
I20260812 03:00:47.641956 20885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:00:47.699117 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:47.684574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.10.118.126:40241)
0812 03:00:47.685330 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:47.685368 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:47.686288 (+ 920us) server_negotiation.cc:407] Connection header received
0812 03:00:47.686757 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:47.686781 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:47.686977 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:47.687118 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0812 03:00:47.688265 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:47.689267 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:47.692227 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:47.692854 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:47.693056 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:47.694144 (+ 1088us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 03:00:47.697654 (+ 3510us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33431/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 03:00:47.698175 (+ 521us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33431/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":172}
W20260812 03:00:47.699373 20869 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33431/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 03:00:47.702904 20905 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:00:47.684284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40241 (local address 127.0.0.1:35642)
0812 03:00:47.686099 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:47.686150 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:47.686453 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:47.687223 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:47.687257 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:47.687313 (+ 56us) client_negotiation.cc:189] Negotiated authn=JWT
0812 03:00:47.688001 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:47.688052 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:47.689531 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:47.689562 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:47.691953 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:47.692005 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:47.693159 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:47.693192 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:47.693614 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:47.693783 (+ 169us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 03:00:47.697976 (+ 4193us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33431/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 03:00:47.701376 (+ 3400us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.118.126:40241: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33431/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":1325,"mutex_wait_us":2818,"thread_start_us":881,"threads_started":1}
I20260812 03:00:47.704679 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 20737
E20260812 03:00:47.732033 20906 webserver.cc:533] Webserver: sslize failed:
E20260812 03:00:47.753221 20906 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T03:00:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (2120 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T03:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:47Z Disabled control of system clock
I20260812 03:00:47.826030 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41215
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:41215
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/testchainca.cert with env {}
W20260812 03:00:48.156236 20912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:48.156873 20912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:48.157411 20912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:48.187649 20912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:48.187943 20912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:48.188105 20912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:48.188304 20912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:48.188510 20912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 03:00:48.188694 20912 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 03:00:48.188832 20912 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 03:00:48.223873 20912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:41215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:48.225008 20912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:48.226226 20912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:48.244710 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:48.244736 20927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:48.245628 20923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:48.247383 20912 server_base.cc:1034] running on GCE node
I20260812 03:00:49.435168 20912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:49.437995 20912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:49.439399 20912 hybrid_clock.cc:648] HybridClock initialized: now 1786503649439372 us; error 73 us; skew 500 ppm
I20260812 03:00:49.440264 20912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:00:49.447423 20912 webserver.cc:460] Webserver started at http://127.10.118.126:33767/ using document root <none> and password file <none>
I20260812 03:00:49.448499 20912 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:49.448742 20912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:49.449267 20912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:49.454677 20912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "a966a56b0ab947409df26b02324825d4"
format_stamp: "Formatted at 2026-08-12 03:00:49 on dist-test-slave-zpvx"
server_key: "75d101368d07c8b505b00e241a8156ce"
server_key_iv: "e019552f68bc99526a2caa38e9ed8279"
server_key_version: "encryptionkey@0"
I20260812 03:00:49.455868 20912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "a966a56b0ab947409df26b02324825d4"
format_stamp: "Formatted at 2026-08-12 03:00:49 on dist-test-slave-zpvx"
server_key: "75d101368d07c8b505b00e241a8156ce"
server_key_iv: "e019552f68bc99526a2caa38e9ed8279"
server_key_version: "encryptionkey@0"
I20260812 03:00:49.463207 20912 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 03:00:49.468987 20947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:49.470150 20912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 03:00:49.470508 20912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "a966a56b0ab947409df26b02324825d4"
format_stamp: "Formatted at 2026-08-12 03:00:49 on dist-test-slave-zpvx"
server_key: "75d101368d07c8b505b00e241a8156ce"
server_key_iv: "e019552f68bc99526a2caa38e9ed8279"
server_key_version: "encryptionkey@0"
I20260812 03:00:49.470897 20912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:49.510926 20912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:49.512403 20912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:49.512863 20912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:49.583173 20912 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:41215
I20260812 03:00:49.583303 21033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:41215 every 8 connection(s)
I20260812 03:00:49.585937 20912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:49.587247 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 20912
I20260812 03:00:49.587718 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:49.589143 10713 external_mini_cluster.cc:1428] Setting key 5ffb2b1ca72de29f2f9a240e30ab7ce4
I20260812 03:00:49.593197 21034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:49.614629 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:49.591537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.10.118.126:41215)
0812 03:00:49.595454 (+ 3917us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:49.595480 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:49.595541 (+ 61us) server_negotiation.cc:407] Connection header received
0812 03:00:49.602433 (+ 6892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:49.602474 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:49.602865 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:49.603292 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:49.604545 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:49.606136 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:49.608105 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:49.608999 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:49.609386 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:49.611127 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:49.611155 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:49.611187 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:00:49.611936 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:49.612028 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:00:49.612074 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:49.612250 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:49.612619 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 03:00:49.612986 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3398,"thread_start_us":2950,"threads_started":1}
I20260812 03:00:49.615665 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4: Bootstrap starting.
I20260812 03:00:49.619014 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:00:49.621491 21034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:49.623309 21034 log.cc:824] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:49.629026 21034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4: No bootstrap required, opened a new log
I20260812 03:00:49.649554 21034 raft_consensus.cc:348] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a966a56b0ab947409df26b02324825d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41215 } }
I20260812 03:00:49.650312 21034 raft_consensus.cc:374] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:49.650560 21034 raft_consensus.cc:729] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a966a56b0ab947409df26b02324825d4, State: Initialized, Role: FOLLOWER
I20260812 03:00:49.651322 21034 consensus_queue.cc:260] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a966a56b0ab947409df26b02324825d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41215 } }
I20260812 03:00:49.652034 21034 raft_consensus.cc:388] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:49.652395 21034 raft_consensus.cc:482] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:49.652789 21034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:49.656884 21034 raft_consensus.cc:504] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a966a56b0ab947409df26b02324825d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41215 } }
I20260812 03:00:49.657646 21034 leader_election.cc:302] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a966a56b0ab947409df26b02324825d4; no voters:
I20260812 03:00:49.659600 21034 leader_election.cc:288] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:49.659978 21041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:49.662787 21041 raft_consensus.cc:686] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [term 1 LEADER]: Becoming Leader. State: Replica: a966a56b0ab947409df26b02324825d4, State: Running, Role: LEADER
I20260812 03:00:49.663537 21041 consensus_queue.cc:237] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a966a56b0ab947409df26b02324825d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41215 } }
I20260812 03:00:49.664523 21034 sys_catalog.cc:564] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:49.675666 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a966a56b0ab947409df26b02324825d4. Latest consensus state: current_term: 1 leader_uuid: "a966a56b0ab947409df26b02324825d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a966a56b0ab947409df26b02324825d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41215 } } }
I20260812 03:00:49.676558 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:49.678241 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a966a56b0ab947409df26b02324825d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a966a56b0ab947409df26b02324825d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41215 } } }
I20260812 03:00:49.679060 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:49.681695 21057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:49.692865 21057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:49.708976 21057 catalog_manager.cc:1347] Generated new cluster ID: 9f84e9d1171c4b68b916c411dcc4f37e
I20260812 03:00:49.709285 21057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:49.722069 21057 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:49.723922 21057 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:49.736079 21057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a966a56b0ab947409df26b02324825d4: Generated new TSK 0
I20260812 03:00:49.737190 21057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:00:50.301646 21039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:49.774415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.10.118.126:41215)
0812 03:00:49.775090 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:49.775127 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:49.776006 (+ 879us) server_negotiation.cc:407] Connection header received
0812 03:00:49.776518 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:49.776536 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:49.776731 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:49.776868 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0812 03:00:49.778039 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:49.778925 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:49.781826 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:49.782382 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:49.782569 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:49.783617 (+ 1048us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 03:00:50.300147 (+516530us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 03:00:50.300703 (+ 556us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44038: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":117}
W20260812 03:00:50.301909 21039 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44038: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 03:00:50.303046 21083 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:00:49.774167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41215 (local address 127.0.0.1:44038)
0812 03:00:49.775833 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0812 03:00:49.775883 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 03:00:49.776215 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:00:49.776967 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:00:49.777000 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:00:49.777057 (+ 57us) client_negotiation.cc:189] Negotiated authn=JWT
0812 03:00:49.777747 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:49.777796 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:49.779173 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:49.779209 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:49.781570 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:00:49.781619 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:00:49.782677 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:00:49.782712 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:00:49.783122 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:49.783293 (+ 171us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 03:00:50.300600 (+517307us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 03:00:50.301168 (+ 568us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.118.126:41215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":838,"threads_started":1}
I20260812 03:00:50.305132 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 20912
2026-08-12T03:00:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (2567 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (2451 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (2471 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:00:55 dist-test-slave-zpvx krb5kdc[21114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:00:55 dist-test-slave-zpvx krb5kdc[21114](info): set up 2 sockets
Aug 12 03:00:55 dist-test-slave-zpvx krb5kdc[21114](info): commencing operation
krb5kdc: starting...
W20260812 03:00:56.606118 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.319s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:00:56 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:00:56.841166 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38629
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:39591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:38629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:57.145203 21138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:57.145864 21138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:57.146526 21138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:57.176498 21138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:00:57.176865 21138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:57.177191 21138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:57.177500 21138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:00:57.177829 21138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:00:57.214530 21138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:39591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:38629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:57.215716 21138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:57.216791 21138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:57.233800 21151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:57.233819 21150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:57.234906 21154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:00:57.236167 21138 server_base.cc:1034] running on GCE node
I20260812 03:00:58.430709 21138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:00:58.433523 21138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:00:58.434880 21138 hybrid_clock.cc:648] HybridClock initialized: now 1786503658434842 us; error 59 us; skew 500 ppm
I20260812 03:00:58.435717 21138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:00:58 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503658, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:00:58.442610 21138 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:00:58.448041 21138 webserver.cc:460] Webserver started at http://127.10.118.126:36989/ using document root <none> and password file <none>
I20260812 03:00:58.449019 21138 fs_manager.cc:359] Metadata directory not provided
I20260812 03:00:58.449265 21138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:00:58.449710 21138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:00:58.454885 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "282e737b9dfe45f39cbad83a029f7ad4"
format_stamp: "Formatted at 2026-08-12 03:00:58 on dist-test-slave-zpvx"
server_key: "75adf4c842bdd2796a2e423dacb950e4"
server_key_iv: "00d6216ab6ff6d38484410d26d35b65e"
server_key_version: "encryptionkey@0"
I20260812 03:00:58.455997 21138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "282e737b9dfe45f39cbad83a029f7ad4"
format_stamp: "Formatted at 2026-08-12 03:00:58 on dist-test-slave-zpvx"
server_key: "75adf4c842bdd2796a2e423dacb950e4"
server_key_iv: "00d6216ab6ff6d38484410d26d35b65e"
server_key_version: "encryptionkey@0"
I20260812 03:00:58.464090 21138 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:00:58.469813 21167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:00:58.470920 21138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260812 03:00:58.471247 21138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "282e737b9dfe45f39cbad83a029f7ad4"
format_stamp: "Formatted at 2026-08-12 03:00:58 on dist-test-slave-zpvx"
server_key: "75adf4c842bdd2796a2e423dacb950e4"
server_key_iv: "00d6216ab6ff6d38484410d26d35b65e"
server_key_version: "encryptionkey@0"
I20260812 03:00:58.471594 21138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:00:58.513689 21138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:00:58.519493 21138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:00:58.519906 21138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:00:58.588804 21138 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:38629
I20260812 03:00:58.588909 21263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:38629 every 8 connection(s)
I20260812 03:00:58.591645 21138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:00:58.597349 21264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:00:58.599568 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 21138
I20260812 03:00:58.600190 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:00:58.601935 10713 external_mini_cluster.cc:1428] Setting key 5f87dee26897f8534004681786937ace
I20260812 03:00:58.626924 21264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4: Bootstrap starting.
Aug 12 03:00:58 dist-test-slave-zpvx krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:00:58.633490 21264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4: Neither blocks nor log segments found. Creating new log.
I20260812 03:00:58.635545 21264 log.cc:824] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4: Log is configured to *not* fsync() on all Append() calls
I20260812 03:00:58.641479 21264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4: No bootstrap required, opened a new log
I20260812 03:00:58.641999 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:00:58.605074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.10.118.126:38629)
0812 03:00:58.607899 (+ 2825us) server_negotiation.cc:206] Beginning negotiation
0812 03:00:58.607926 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:00:58.607982 (+ 56us) server_negotiation.cc:407] Connection header received
0812 03:00:58.612043 (+ 4061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:00:58.612088 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:00:58.612535 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:00:58.612907 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:00:58.616589 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:58.618281 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:58.621171 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:00:58.622053 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:00:58.622299 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:00:58.629938 (+ 7639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:00:58.629963 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:00:58.629993 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:00:58.630093 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:00:58.634361 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:58.635869 (+ 1508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:58.635905 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:58.635922 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:58.636162 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:00:58.636994 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:00:58.637041 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:00:58.637081 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:00:58.638276 (+ 1195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:00:58.638824 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0812 03:00:58.639422 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 03:00:58.639959 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2393,"thread_start_us":1882,"threads_started":1}
I20260812 03:00:58.661700 21264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282e737b9dfe45f39cbad83a029f7ad4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38629 } }
I20260812 03:00:58.662588 21264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:00:58.662839 21264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 282e737b9dfe45f39cbad83a029f7ad4, State: Initialized, Role: FOLLOWER
I20260812 03:00:58.663569 21264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282e737b9dfe45f39cbad83a029f7ad4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38629 } }
I20260812 03:00:58.664099 21264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:00:58.664426 21264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:00:58.664769 21264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:00:58.668890 21264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282e737b9dfe45f39cbad83a029f7ad4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38629 } }
I20260812 03:00:58.669705 21264 leader_election.cc:302] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 282e737b9dfe45f39cbad83a029f7ad4; no voters:
I20260812 03:00:58.671981 21264 leader_election.cc:288] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:00:58.672436 21274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:00:58.674872 21274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [term 1 LEADER]: Becoming Leader. State: Replica: 282e737b9dfe45f39cbad83a029f7ad4, State: Running, Role: LEADER
I20260812 03:00:58.675782 21274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282e737b9dfe45f39cbad83a029f7ad4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38629 } }
I20260812 03:00:58.677170 21264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:00:58.686269 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 282e737b9dfe45f39cbad83a029f7ad4. Latest consensus state: current_term: 1 leader_uuid: "282e737b9dfe45f39cbad83a029f7ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282e737b9dfe45f39cbad83a029f7ad4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38629 } } }
I20260812 03:00:58.687192 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:58.692008 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "282e737b9dfe45f39cbad83a029f7ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "282e737b9dfe45f39cbad83a029f7ad4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38629 } } }
I20260812 03:00:58.694820 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:00:58.700090 21283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:00:58.711309 21283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:00:58.729893 21283 catalog_manager.cc:1347] Generated new cluster ID: c80d7e51a56d489f852b595e81f53e6e
I20260812 03:00:58.730178 21283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:00:58.753662 21283 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:00:58.755429 21283 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:00:58.776077 21283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 282e737b9dfe45f39cbad83a029f7ad4: Generated new TSK 0
I20260812 03:00:58.777276 21283 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:00:58.837142 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38629
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:39591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:00:59.154488 21308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:00:59.155027 21308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:00:59.155736 21308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:00:59.186811 21308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:00:59.187224 21308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:00:59.188171 21308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:00:59.226972 21308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:39591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:00:59.228201 21308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:00:59.229303 21308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:00:59.250751 21320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:59.251354 21322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:00:59.250706 21319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:00.532523 21321 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:01:00.532816 21308 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:01:00.536587 21308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:00.539373 21308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:00.540797 21308 hybrid_clock.cc:648] HybridClock initialized: now 1786503660540770 us; error 50 us; skew 500 ppm
I20260812 03:01:00.541581 21308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:00 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503660, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:00.552644 21308 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:01:00.558019 21308 webserver.cc:460] Webserver started at http://127.10.118.65:36063/ using document root <none> and password file <none>
I20260812 03:01:00.558980 21308 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:00.559221 21308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:00.559693 21308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:00.565121 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "3b8103b6f50042f590564bc13d12fbed"
format_stamp: "Formatted at 2026-08-12 03:01:00 on dist-test-slave-zpvx"
server_key: "5303b695e552ae38cae04047b1904ab5"
server_key_iv: "a45504ed881ef10222b1f1a5225544fd"
server_key_version: "encryptionkey@0"
I20260812 03:01:00.566304 21308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "3b8103b6f50042f590564bc13d12fbed"
format_stamp: "Formatted at 2026-08-12 03:01:00 on dist-test-slave-zpvx"
server_key: "5303b695e552ae38cae04047b1904ab5"
server_key_iv: "a45504ed881ef10222b1f1a5225544fd"
server_key_version: "encryptionkey@0"
I20260812 03:01:00.573916 21308 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 03:01:00.580178 21337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:00.581382 21308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 03:01:00.581657 21308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "3b8103b6f50042f590564bc13d12fbed"
format_stamp: "Formatted at 2026-08-12 03:01:00 on dist-test-slave-zpvx"
server_key: "5303b695e552ae38cae04047b1904ab5"
server_key_iv: "a45504ed881ef10222b1f1a5225544fd"
server_key_version: "encryptionkey@0"
I20260812 03:01:00.582000 21308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:00.630715 21308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:00.636648 21308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:00.637071 21308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:00.639570 21308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:00.644007 21308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:00.644200 21308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:00.644480 21308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:00.644657 21308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:00.784415 21308 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:36991
I20260812 03:01:00.784579 21544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:36991 every 8 connection(s)
I20260812 03:01:00.787046 21308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:01:00.791903 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 21308
I20260812 03:01:00.792510 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:01:00.794262 10713 external_mini_cluster.cc:1428] Setting key 79299cbfcf788412e0ca6a6d9bba609f
Aug 12 03:01:00 dist-test-slave-zpvx krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503660, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:01:00.824198 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:00.791972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:50481 (local address 127.10.118.126:38629)
0812 03:01:00.793646 (+ 1674us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:00.793669 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:00.795103 (+ 1434us) server_negotiation.cc:407] Connection header received
0812 03:01:00.800789 (+ 5686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:00.800829 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:00.801004 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:00.801182 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:00.803749 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:00.804789 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:00.806855 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:00.807584 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:00.807881 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:00.813222 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:00.813246 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:00.813262 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:00.813356 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:00.817181 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:00.818409 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:00.818430 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:00.818447 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:00.818642 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:00.819427 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:00.819446 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:00.819460 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:00.820067 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:00.820317 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:00.821438 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0812 03:01:00.821927 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":858,"threads_started":1}
I20260812 03:01:00.825032 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:00.792024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38629 (local address 127.10.118.65:50481)
0812 03:01:00.794867 (+ 2843us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:00.794936 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:00.800443 (+ 5507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:00.801476 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:00.801512 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:00.802452 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:00.803460 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:00.803504 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:00.805043 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:00.805064 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:00.806596 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:00.806636 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:00.807950 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:00.807988 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:00.808564 (+ 576us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:00.809458 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:00.809522 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:00.812858 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:00.817419 (+ 4561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:00.817459 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:00.817484 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:00.818198 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:00.818844 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:00.818903 (+ 59us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:00.818930 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:00.819221 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:00.820282 (+ 1061us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:00.820326 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:00.820830 (+ 504us) client_negotiation.cc:769] Sending connection context
0812 03:01:00.821210 (+ 380us) client_negotiation.cc:240] Negotiation successful
0812 03:01:00.821935 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2327,"thread_start_us":1618,"threads_started":1}
I20260812 03:01:00.827086 21545 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38629
I20260812 03:01:00.827514 21545 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:00.828521 21545 heartbeater.cc:511] Master 127.10.118.126:38629 requested a full tablet report, sending...
I20260812 03:01:00.831236 21195 ts_manager.cc:194] Registered new tserver with Master: 3b8103b6f50042f590564bc13d12fbed (127.10.118.65:36991)
I20260812 03:01:00.833324 21195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:50481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:01:00.852470 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38629
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:39591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:01.169364 21553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:01.169881 21553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:01.170478 21553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:01.204425 21553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:01.204854 21553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:01.205765 21553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:01:01.242007 21553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:39591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:01.243192 21553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:01.244235 21553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:01.265715 21564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:01.837337 21545 heartbeater.cc:503] Master 127.10.118.126:38629 was elected leader, sending a full tablet report...
W20260812 03:01:01.265795 21571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:01.268209 21553 server_base.cc:1034] running on GCE node
W20260812 03:01:01.265803 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:02.448282 21553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:02.450673 21553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:02.452031 21553 hybrid_clock.cc:648] HybridClock initialized: now 1786503662451995 us; error 63 us; skew 500 ppm
I20260812 03:01:02.453101 21553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:02 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503662, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:02.459281 21553 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:01:02.464663 21553 webserver.cc:460] Webserver started at http://127.10.118.66:45423/ using document root <none> and password file <none>
I20260812 03:01:02.465605 21553 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:02.465842 21553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:02.466306 21553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:02.471932 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "f4a9c72ab78c4f26a4306ebf92f4b741"
format_stamp: "Formatted at 2026-08-12 03:01:02 on dist-test-slave-zpvx"
server_key: "3971f9bd74da3d9f112add7113452792"
server_key_iv: "8f56255375a0722c349cf48485b72436"
server_key_version: "encryptionkey@0"
I20260812 03:01:02.473404 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "f4a9c72ab78c4f26a4306ebf92f4b741"
format_stamp: "Formatted at 2026-08-12 03:01:02 on dist-test-slave-zpvx"
server_key: "3971f9bd74da3d9f112add7113452792"
server_key_iv: "8f56255375a0722c349cf48485b72436"
server_key_version: "encryptionkey@0"
I20260812 03:01:02.481297 21553 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:01:02.487000 21585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:02.488188 21553 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 03:01:02.488556 21553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "f4a9c72ab78c4f26a4306ebf92f4b741"
format_stamp: "Formatted at 2026-08-12 03:01:02 on dist-test-slave-zpvx"
server_key: "3971f9bd74da3d9f112add7113452792"
server_key_iv: "8f56255375a0722c349cf48485b72436"
server_key_version: "encryptionkey@0"
I20260812 03:01:02.488951 21553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:02.531222 21553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:02.536743 21553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:02.537214 21553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:02.539834 21553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:02.543840 21553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:02.544065 21553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:02.544348 21553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:02.544546 21553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 03:01:02.690521 21553 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:43251
I20260812 03:01:02.690732 21786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:43251 every 8 connection(s)
I20260812 03:01:02.694288 21553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:01:02.703629 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 21553
I20260812 03:01:02.704098 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:01:02.705200 10713 external_mini_cluster.cc:1428] Setting key 135bd3975ef017b53b00f75b396f0db8
Aug 12 03:01:02 dist-test-slave-zpvx krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503662, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:01:02.731001 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:02.698988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:53757 (local address 127.10.118.126:38629)
0812 03:01:02.700125 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:02.700158 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:02.702315 (+ 2157us) server_negotiation.cc:407] Connection header received
0812 03:01:02.707347 (+ 5032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:02.707378 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:02.707595 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:02.707783 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:02.710607 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:02.711513 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:02.713698 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:02.714298 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:02.714593 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:02.720209 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:02.720233 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:02.720249 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:02.720351 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:02.724058 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:02.725746 (+ 1688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:02.725766 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:02.725782 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:02.725979 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:02.726895 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:02.726914 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:02.726927 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:02.727408 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:02.727586 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:02.729106 (+ 1520us) server_negotiation.cc:299] Negotiation successful
0812 03:01:02.729442 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":780,"thread_start_us":502,"threads_started":1}
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
I20260812 03:01:02.732606 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:02.699271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38629 (local address 127.10.118.66:53757)
0812 03:01:02.702078 (+ 2807us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:02.702155 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:02.706977 (+ 4822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:02.708042 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:02.708090 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:02.709346 (+ 1256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:02.710314 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:02.710357 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:02.711779 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:02.711800 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:02.713430 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:02.713469 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:02.714630 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:02.714668 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:02.715139 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:02.716042 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:02.716113 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:02.719855 (+ 3742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:02.724287 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:02.724328 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:02.724397 (+ 69us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:02.725520 (+ 1123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:02.726186 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:02.726223 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:02.726255 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:02.726677 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:02.727649 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:02.727701 (+ 52us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:02.728419 (+ 718us) client_negotiation.cc:769] Sending connection context
0812 03:01:02.728916 (+ 497us) client_negotiation.cc:240] Negotiation successful
0812 03:01:02.729693 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2239,"thread_start_us":1477,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:01:02.734853 21788 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38629
I20260812 03:01:02.735314 21788 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:02.736240 21788 heartbeater.cc:511] Master 127.10.118.126:38629 requested a full tablet report, sending...
I20260812 03:01:02.738430 21195 ts_manager.cc:194] Registered new tserver with Master: f4a9c72ab78c4f26a4306ebf92f4b741 (127.10.118.66:43251)
I20260812 03:01:02.739857 21195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:53757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:01:02.764561 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38629
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:39591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:03.077140 21802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:03.077725 21802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:03.078379 21802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:03.109712 21802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:03.110165 21802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:03.111099 21802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:01:03.146062 21802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:39591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:03.147284 21802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:03.148423 21802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:03.168294 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:03.744181 21788 heartbeater.cc:503] Master 127.10.118.126:38629 was elected leader, sending a full tablet report...
W20260812 03:01:03.168363 21813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:03.168803 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:03.169708 21802 server_base.cc:1034] running on GCE node
I20260812 03:01:04.362084 21802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:04.365034 21802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:04.366506 21802 hybrid_clock.cc:648] HybridClock initialized: now 1786503664366439 us; error 82 us; skew 500 ppm
I20260812 03:01:04.367492 21802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.373570 21802 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:01:04.379480 21802 webserver.cc:460] Webserver started at http://127.10.118.67:38919/ using document root <none> and password file <none>
I20260812 03:01:04.380473 21802 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:04.380666 21802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:04.381083 21802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:04.386287 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "069d4b88b6614ec3ad8cde166d4ff1bf"
format_stamp: "Formatted at 2026-08-12 03:01:04 on dist-test-slave-zpvx"
server_key: "6b2dddcf618e30a19d8a18dee6491d8e"
server_key_iv: "877ef91759f3201d029bf3051b1bc7e8"
server_key_version: "encryptionkey@0"
I20260812 03:01:04.387432 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "069d4b88b6614ec3ad8cde166d4ff1bf"
format_stamp: "Formatted at 2026-08-12 03:01:04 on dist-test-slave-zpvx"
server_key: "6b2dddcf618e30a19d8a18dee6491d8e"
server_key_iv: "877ef91759f3201d029bf3051b1bc7e8"
server_key_version: "encryptionkey@0"
I20260812 03:01:04.395442 21802 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20260812 03:01:04.400964 21835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:04.402070 21802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 03:01:04.402410 21802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "069d4b88b6614ec3ad8cde166d4ff1bf"
format_stamp: "Formatted at 2026-08-12 03:01:04 on dist-test-slave-zpvx"
server_key: "6b2dddcf618e30a19d8a18dee6491d8e"
server_key_iv: "877ef91759f3201d029bf3051b1bc7e8"
server_key_version: "encryptionkey@0"
I20260812 03:01:04.402856 21802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:04.436041 21802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:04.441679 21802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:04.442157 21802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:04.444779 21802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:04.448881 21802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:04.449146 21802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:04.449409 21802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:04.449577 21802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:04.587221 21802 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:38563
I20260812 03:01:04.587397 22016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:38563 every 8 connection(s)
I20260812 03:01:04.590035 21802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:01:04.591550 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 21802
I20260812 03:01:04.592558 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:01:04.595795 10713 external_mini_cluster.cc:1428] Setting key 4107f7e54ba41a8bb7a032f4cc6337a4
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:01:04.630352 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:04.595377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:57791 (local address 127.10.118.126:38629)
0812 03:01:04.598513 (+ 3136us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:04.598535 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:04.600550 (+ 2015us) server_negotiation.cc:407] Connection header received
0812 03:01:04.605647 (+ 5097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:04.605680 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:04.605896 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:04.606072 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:04.608727 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.610177 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.612322 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.613323 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.613599 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:04.619375 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:04.619399 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:04.619415 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:04.619513 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:04.623387 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.624768 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.624789 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.624803 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.624996 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.625720 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.625738 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.625751 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.626244 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:04.626443 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:04.627748 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0812 03:01:04.628305 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2628,"thread_start_us":2344,"threads_started":1}
I20260812 03:01:04.631197 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:04.595819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38629 (local address 127.10.118.67:57791)
0812 03:01:04.600262 (+ 4443us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:04.600333 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:04.605304 (+ 4971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:04.606308 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:04.606332 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:04.607167 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:04.608260 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:04.608306 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.610465 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.610491 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:04.612057 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:04.612097 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.613670 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.613707 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:04.614136 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:04.615241 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:04.615312 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:04.619038 (+ 3726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:04.623592 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:04.623616 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:04.623644 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:04.624569 (+ 925us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:04.625189 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:04.625210 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:04.625228 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:04.625535 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:04.626448 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:04.626479 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:04.626953 (+ 474us) client_negotiation.cc:769] Sending connection context
0812 03:01:04.627454 (+ 501us) client_negotiation.cc:240] Negotiation successful
0812 03:01:04.628150 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3847,"thread_start_us":2961,"threads_started":1}
I20260812 03:01:04.633018 22020 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38629
I20260812 03:01:04.633379 22020 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:04.634253 22020 heartbeater.cc:511] Master 127.10.118.126:38629 requested a full tablet report, sending...
I20260812 03:01:04.636251 21195 ts_manager.cc:194] Registered new tserver with Master: 069d4b88b6614ec3ad8cde166d4ff1bf (127.10.118.67:38563)
I20260812 03:01:04.637745 21195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:57791
I20260812 03:01:04.644979 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/test-admin.keytab.
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.683652 10713 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 03:01:04.698622 22038 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:04.693140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.10.118.65:36991)
0812 03:01:04.695240 (+ 2100us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:04.695270 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:04.695313 (+ 43us) server_negotiation.cc:407] Connection header received
0812 03:01:04.695428 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:04.695446 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:04.695693 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:04.695859 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:04.697783 (+ 1924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43380: BlockingRecv error: recv got EOF from 127.0.0.1:43380 (error 108)
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":1311,"threads_started":1}
W20260812 03:01:04.698832 10713 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5cc' of type FILE
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.701900 10713 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.703882 10713 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 03:01:04.728543 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:04.706786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.10.118.65:36991)
0812 03:01:04.707329 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:04.707357 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:04.707795 (+ 438us) server_negotiation.cc:407] Connection header received
0812 03:01:04.708067 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:04.708089 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:04.708257 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:04.708468 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:04.710181 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.711531 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.713435 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.714199 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.714400 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:04.719175 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:04.719201 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:04.719232 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:04.719327 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:04.723405 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.724520 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.724542 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.724556 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.724780 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.725360 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.725378 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.725392 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.725930 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:04.726280 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:04.726853 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 03:01:04.727179 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260812 03:01:04.738348 10713 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5cc' of type FILE
W20260812 03:01:04.738669 22038 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:04.734942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.10.118.65:36991)
0812 03:01:04.735317 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:04.735337 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:04.736121 (+ 784us) server_negotiation.cc:407] Connection header received
0812 03:01:04.736429 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:04.736461 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:04.736616 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:04.736743 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:04.737703 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43396: BlockingRecv error: recv got EOF from 127.0.0.1:43396 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.740433 10713 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.741991 10713 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 03:01:04.764477 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:04.744652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.10.118.65:36991)
0812 03:01:04.745138 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:04.745167 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:04.745760 (+ 593us) server_negotiation.cc:407] Connection header received
0812 03:01:04.746101 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:04.746120 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:04.746281 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:04.746437 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:04.747958 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.748873 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.750609 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.751170 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.751418 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:04.755902 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:04.755922 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:04.755938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:04.756036 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:04.759543 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.760584 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.760607 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.760620 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.760816 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.761356 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.761375 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.761388 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.761864 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:04.762119 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:04.762656 (+ 537us) server_negotiation.cc:299] Negotiation successful
0812 03:01:04.763031 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260812 03:01:04.772848 10713 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786503542438041-10713-0/krb5kdc/krb5cc' of type FILE
W20260812 03:01:04.773192 22038 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:04.769627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.10.118.65:36991)
0812 03:01:04.770175 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:04.770200 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:04.770566 (+ 366us) server_negotiation.cc:407] Connection header received
0812 03:01:04.770820 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:04.770837 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:04.770995 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:04.771119 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:04.772272 (+ 1153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43420: BlockingRecv error: recv got EOF from 127.0.0.1:43420 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.774868 10713 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:04.776449 10713 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 03:01:04 dist-test-slave-zpvx krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.65@KRBTEST.COM
I20260812 03:01:04.802896 22038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:04.779209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.10.118.65:36991)
0812 03:01:04.779567 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:04.779585 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:04.780555 (+ 970us) server_negotiation.cc:407] Connection header received
0812 03:01:04.780834 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:04.780851 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:04.781008 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:04.781135 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:04.782621 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.784019 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.785835 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:04.786793 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:04.787085 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:04.791734 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:04.791778 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:04.791805 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:04.791939 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:04.796686 (+ 4747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.797786 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.797836 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.797875 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.798234 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:04.798856 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:04.798884 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:04.798904 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:04.799516 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:04.799868 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:04.800417 (+ 549us) server_negotiation.cc:299] Negotiation successful
0812 03:01:04.800855 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 03:01:04.805989 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 21308
I20260812 03:01:04.844746 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 21553
I20260812 03:01:04.879745 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 21802
I20260812 03:01:04.917318 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 21138
2026-08-12T03:01:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (9699 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:877: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:01:05 dist-test-slave-zpvx krb5kdc[22061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:01:05 dist-test-slave-zpvx krb5kdc[22061](info): set up 2 sockets
Aug 12 03:01:05 dist-test-slave-zpvx krb5kdc[22061](info): commencing operation
krb5kdc: starting...
W20260812 03:01:06.305512 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:01:06 dist-test-slave-zpvx krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:01:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.126.keytab.
Entry for principal oryx/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.126.keytab.
I20260812 03:01:06.532826 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43757
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.126.keytab
--principal=oryx/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:43757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:06.838210 22089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:06.838840 22089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:06.839457 22089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:06.869426 22089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:01:06.869755 22089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 03:01:06.870018 22089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:06.870339 22089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:06.870596 22089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:01:06.870857 22089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:01:06.905983 22089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:43757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.126.keytab
--principal=oryx/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:06.907204 22089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:06.908262 22089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:06.924573 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:06.924587 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:06.924628 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:06.926466 22089 server_base.cc:1034] running on GCE node
I20260812 03:01:08.114004 22089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:08.116802 22089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:08.118193 22089 hybrid_clock.cc:648] HybridClock initialized: now 1786503668118173 us; error 61 us; skew 500 ppm
I20260812 03:01:08.119055 22089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:08 dist-test-slave-zpvx krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:08.130072 22089 init.cc:377] Logged in from keytab as oryx/127.10.118.126@KRBTEST.COM (short username oryx)
I20260812 03:01:08.135859 22089 webserver.cc:460] Webserver started at http://127.10.118.126:34931/ using document root <none> and password file <none>
I20260812 03:01:08.136847 22089 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:08.137080 22089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:08.137549 22089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:08.142735 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21"
format_stamp: "Formatted at 2026-08-12 03:01:08 on dist-test-slave-zpvx"
server_key: "994af62b07d1ff2798271ce624f308b0"
server_key_iv: "e5418ae450f6400589bb3dd7d9bef63c"
server_key_version: "encryptionkey@0"
I20260812 03:01:08.143967 22089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21"
format_stamp: "Formatted at 2026-08-12 03:01:08 on dist-test-slave-zpvx"
server_key: "994af62b07d1ff2798271ce624f308b0"
server_key_iv: "e5418ae450f6400589bb3dd7d9bef63c"
server_key_version: "encryptionkey@0"
I20260812 03:01:08.151300 22089 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 03:01:08.156945 22121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:08.158002 22089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 03:01:08.158341 22089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21"
format_stamp: "Formatted at 2026-08-12 03:01:08 on dist-test-slave-zpvx"
server_key: "994af62b07d1ff2798271ce624f308b0"
server_key_iv: "e5418ae450f6400589bb3dd7d9bef63c"
server_key_version: "encryptionkey@0"
I20260812 03:01:08.158713 22089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:08.200238 22089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:08.206219 22089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:08.206707 22089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:08.308827 22089 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:43757
I20260812 03:01:08.308936 22224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:43757 every 8 connection(s)
I20260812 03:01:08.311650 22089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:01:08.317034 22227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:08.319948 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 22089
I20260812 03:01:08.320422 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:01:08.321587 10713 external_mini_cluster.cc:1428] Setting key b360dc012dfbd50db20d36cc0ed9229a
I20260812 03:01:08.338656 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Bootstrap starting.
I20260812 03:01:08.345808 22227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Neither blocks nor log segments found. Creating new log.
Aug 12 03:01:08 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM
I20260812 03:01:08.347509 22227 log.cc:824] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Log is configured to *not* fsync() on all Append() calls
I20260812 03:01:08.352741 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: No bootstrap required, opened a new log
I20260812 03:01:08.359151 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:08.324235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.10.118.126:43757)
0812 03:01:08.326983 (+ 2748us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:08.327010 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:08.327063 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:01:08.331015 (+ 3952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:08.331057 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:08.331401 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:08.331713 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:08.334987 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:08.336768 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:08.339444 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:08.340301 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:08.340608 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:08.347426 (+ 6818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:08.347449 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:08.347478 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:08.347580 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:08.352563 (+ 4983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:08.353714 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:08.353739 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:08.353756 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:08.353990 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:08.354636 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:08.354659 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:08.354672 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:08.355356 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:08.355826 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:08.356579 (+ 753us) server_negotiation.cc:299] Negotiation successful
0812 03:01:08.357194 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2320,"thread_start_us":1819,"threads_started":1}
I20260812 03:01:08.374130 22227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43757 } }
I20260812 03:01:08.374854 22227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:08.375145 22227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebb6d41de8f4ee2af6d9cf7fc163f21, State: Initialized, Role: FOLLOWER
I20260812 03:01:08.375916 22227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43757 } }
I20260812 03:01:08.376626 22227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:01:08.376958 22227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:01:08.377369 22227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:08.381760 22227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43757 } }
I20260812 03:01:08.382484 22227 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ebb6d41de8f4ee2af6d9cf7fc163f21; no voters:
I20260812 03:01:08.384291 22227 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:01:08.384779 22234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:08.387320 22234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [term 1 LEADER]: Becoming Leader. State: Replica: 4ebb6d41de8f4ee2af6d9cf7fc163f21, State: Running, Role: LEADER
I20260812 03:01:08.388098 22234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43757 } }
I20260812 03:01:08.389035 22227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:01:08.399053 22241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebb6d41de8f4ee2af6d9cf7fc163f21. Latest consensus state: current_term: 1 leader_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43757 } } }
I20260812 03:01:08.400027 22241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:08.399667 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebb6d41de8f4ee2af6d9cf7fc163f21" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43757 } } }
I20260812 03:01:08.400556 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:08.407313 22261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:01:08.418104 22261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:01:08.435299 22261 catalog_manager.cc:1347] Generated new cluster ID: f3536ea07b034399abdb4d9a87c57503
I20260812 03:01:08.435712 22261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:01:08.464598 22261 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:01:08.466665 22261 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:01:08.485848 22261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Generated new TSK 0
I20260812 03:01:08.486979 22261 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab.
Entry for principal oryx/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab.
I20260812 03:01:08.548206 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43757
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab
--principal=oryx/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:08.864153 22283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:08.864755 22283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:08.865430 22283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:08.896695 22283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:08.897078 22283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:08.897742 22283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 03:01:08.898259 22283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:01:08.934417 22283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab
--principal=oryx/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:08.935597 22283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:08.936748 22283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:08.957440 22297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:08.958401 22296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:08.958148 22299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:10.461845 22298 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1501 milliseconds
I20260812 03:01:10.461995 22283 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:01:10.463372 22283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:10.466538 22283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:10.468020 22283 hybrid_clock.cc:648] HybridClock initialized: now 1786503670467992 us; error 48 us; skew 500 ppm
I20260812 03:01:10.469218 22283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:10 dist-test-slave-zpvx krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503670, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:10.476143 22283 init.cc:377] Logged in from keytab as oryx/127.10.118.65@KRBTEST.COM (short username oryx)
I20260812 03:01:10.481941 22283 webserver.cc:460] Webserver started at http://127.10.118.65:44683/ using document root <none> and password file <none>
I20260812 03:01:10.482980 22283 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:10.483233 22283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:10.483781 22283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:10.489540 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "8d09f723a9a2408c8a3c98a88bb8ca56"
format_stamp: "Formatted at 2026-08-12 03:01:10 on dist-test-slave-zpvx"
server_key: "0a1244bc19167c7b026133c15b32a337"
server_key_iv: "62fda40e5cd3463fa0c509740cf4e1bc"
server_key_version: "encryptionkey@0"
I20260812 03:01:10.490995 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "8d09f723a9a2408c8a3c98a88bb8ca56"
format_stamp: "Formatted at 2026-08-12 03:01:10 on dist-test-slave-zpvx"
server_key: "0a1244bc19167c7b026133c15b32a337"
server_key_iv: "62fda40e5cd3463fa0c509740cf4e1bc"
server_key_version: "encryptionkey@0"
I20260812 03:01:10.498875 22283 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 03:01:10.504819 22314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:10.506036 22283 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:01:10.506394 22283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "8d09f723a9a2408c8a3c98a88bb8ca56"
format_stamp: "Formatted at 2026-08-12 03:01:10 on dist-test-slave-zpvx"
server_key: "0a1244bc19167c7b026133c15b32a337"
server_key_iv: "62fda40e5cd3463fa0c509740cf4e1bc"
server_key_version: "encryptionkey@0"
I20260812 03:01:10.506827 22283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:10.546833 22283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:10.552794 22283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:10.553290 22283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:10.564286 22283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:10.564659 22283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 03:01:10.565001 22283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:10.565207 22283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 03:01:10 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503670, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM
I20260812 03:01:10.596997 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:10.560628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:56955 (local address 127.10.118.126:43757)
0812 03:01:10.561800 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:10.561833 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:10.565131 (+ 3298us) server_negotiation.cc:407] Connection header received
0812 03:01:10.570134 (+ 5003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:10.570166 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:10.570469 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:10.570668 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:10.573081 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.574016 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.576134 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.576804 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.577088 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:10.583219 (+ 6131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:10.583243 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:10.583259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:10.583354 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:10.588186 (+ 4832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:10.589840 (+ 1654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:10.589890 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:10.589933 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:10.590309 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:10.591396 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:10.591451 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:10.591489 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:10.592411 (+ 922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:10.592755 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:10.594145 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0812 03:01:10.594658 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":569,"threads_started":1}
I20260812 03:01:10.597819 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:10.560987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43757 (local address 127.10.118.65:56955)
0812 03:01:10.564881 (+ 3894us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:10.564968 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:10.569765 (+ 4797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:10.570960 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:10.570999 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:10.571836 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:10.572806 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:10.572851 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.574288 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.574310 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:10.575876 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:10.575927 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.577191 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.577248 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:10.577881 (+ 633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:10.578991 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:10.579055 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:10.582872 (+ 3817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:10.588453 (+ 5581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:10.588481 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:10.588504 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:10.589478 (+ 974us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:10.590540 (+ 1062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:10.590559 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:10.590577 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:10.591087 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:10.592722 (+ 1635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:10.592788 (+ 66us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:10.593512 (+ 724us) client_negotiation.cc:769] Sending connection context
0812 03:01:10.593915 (+ 403us) client_negotiation.cc:240] Negotiation successful
0812 03:01:10.594682 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3348,"thread_start_us":2446,"threads_started":1}
I20260812 03:01:10.644291 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:10.615028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:54503 (local address 127.10.118.126:43757)
0812 03:01:10.615694 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:10.615734 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:10.619051 (+ 3317us) server_negotiation.cc:407] Connection header received
0812 03:01:10.619427 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:10.619467 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:10.619764 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:10.620026 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:10.622849 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.624748 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.626925 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.627642 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.627958 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:10.634024 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:10.634061 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:10.634091 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:10.634237 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:10.637932 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:10.639019 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:10.639039 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:10.639052 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:10.639247 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:10.639890 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:10.639909 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:10.639922 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:10.640502 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:10.640744 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:10.641538 (+ 794us) server_negotiation.cc:299] Negotiation successful
0812 03:01:10.642128 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260812 03:01:10.644680 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:10.615003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43757 (local address 127.10.118.65:54503)
0812 03:01:10.618837 (+ 3834us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:10.618881 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:10.619117 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:10.620092 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:10.620122 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:10.621402 (+ 1280us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:10.622437 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:10.622511 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.625051 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.625090 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:10.626659 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:10.626700 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:10.627968 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:10.627999 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:10.628528 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:10.629560 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:10.629650 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:10.633581 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:10.638172 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:10.638212 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:10.638227 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:10.638802 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:10.639431 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:10.639468 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:10.639482 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:10.639687 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:10.640692 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:10.640720 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:10.641044 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 03:01:10.641312 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 03:01:10.641882 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3318,"thread_start_us":2858,"threads_started":1}
I20260812 03:01:10.743366 22283 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:42009
I20260812 03:01:10.744001 22519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:42009 every 8 connection(s)
I20260812 03:01:10.746201 22283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:01:10.750769 22520 heartbeater.cc:347] Connected to a master server at 127.10.118.126:43757
I20260812 03:01:10.751639 22520 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:10.752753 22520 heartbeater.cc:511] Master 127.10.118.126:43757 requested a full tablet report, sending...
I20260812 03:01:10.754752 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 22283
I20260812 03:01:10.755329 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:01:10.755736 22150 ts_manager.cc:194] Registered new tserver with Master: 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009)
I20260812 03:01:10.756922 10713 external_mini_cluster.cc:1428] Setting key 20386e96333c5651284b19eb7118891d
I20260812 03:01:10.758589 22150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:56955
WARNING: no policy specified for oryx/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.118.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab.
Entry for principal oryx/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab.
I20260812 03:01:10.812306 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43757
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab
--principal=oryx/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:11.124114 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:11.124691 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:11.125370 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:11.156306 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:11.156736 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:11.157392 22526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 03:01:11.157907 22526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:01:11.193131 22526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab
--principal=oryx/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:11.194311 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:11.195359 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:11.215296 22535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:11.763144 22520 heartbeater.cc:503] Master 127.10.118.126:43757 was elected leader, sending a full tablet report...
W20260812 03:01:11.215390 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:11.216226 22534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:12.381709 22538 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:01:12.381851 22526 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:01:12.386775 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:12.389310 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:12.390753 22526 hybrid_clock.cc:648] HybridClock initialized: now 1786503672390712 us; error 36 us; skew 500 ppm
I20260812 03:01:12.391719 22526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:12 dist-test-slave-zpvx krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503672, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:12.398365 22526 init.cc:377] Logged in from keytab as oryx/127.10.118.66@KRBTEST.COM (short username oryx)
I20260812 03:01:12.403596 22526 webserver.cc:460] Webserver started at http://127.10.118.66:41479/ using document root <none> and password file <none>
I20260812 03:01:12.404706 22526 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:12.404970 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:12.405494 22526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:12.411448 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "f04caa54d700475697a9dcfe72b8d7b4"
format_stamp: "Formatted at 2026-08-12 03:01:12 on dist-test-slave-zpvx"
server_key: "ffb2431e0ce32783d9f66e722be64d77"
server_key_iv: "b4413b10bb54bf620fd7f641bd7a665e"
server_key_version: "encryptionkey@0"
I20260812 03:01:12.412856 22526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "f04caa54d700475697a9dcfe72b8d7b4"
format_stamp: "Formatted at 2026-08-12 03:01:12 on dist-test-slave-zpvx"
server_key: "ffb2431e0ce32783d9f66e722be64d77"
server_key_iv: "b4413b10bb54bf620fd7f641bd7a665e"
server_key_version: "encryptionkey@0"
I20260812 03:01:12.420984 22526 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:01:12.427109 22556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:12.428267 22526 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:01:12.428658 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "f04caa54d700475697a9dcfe72b8d7b4"
format_stamp: "Formatted at 2026-08-12 03:01:12 on dist-test-slave-zpvx"
server_key: "ffb2431e0ce32783d9f66e722be64d77"
server_key_iv: "b4413b10bb54bf620fd7f641bd7a665e"
server_key_version: "encryptionkey@0"
I20260812 03:01:12.429090 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:12.466385 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:12.472247 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:12.472777 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:12.485009 22526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:12.485335 22526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:12.485611 22526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:12.485785 22526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 03:01:12 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503672, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM
I20260812 03:01:12.517868 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:12.479592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:48067 (local address 127.10.118.126:43757)
0812 03:01:12.480877 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:12.480892 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:12.483714 (+ 2822us) server_negotiation.cc:407] Connection header received
0812 03:01:12.489760 (+ 6046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:12.489799 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:12.489973 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:12.490192 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:12.492604 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.493533 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.496609 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.497250 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.497542 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:12.503503 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:12.503524 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:12.503573 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:12.503698 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:12.508929 (+ 5231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:12.510821 (+ 1892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:12.510851 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:12.510880 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:12.511160 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:12.512233 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:12.512262 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:12.512284 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:12.513120 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:12.513453 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:12.515022 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0812 03:01:12.515532 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":749,"threads_started":1}
I20260812 03:01:12.518676 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:12.479925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43757 (local address 127.10.118.66:48067)
0812 03:01:12.483469 (+ 3544us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:12.483541 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:12.489404 (+ 5863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:12.490484 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:12.490526 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:12.491364 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:12.492266 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:12.492310 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.493808 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.493828 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:12.496249 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:12.496305 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.497591 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.497628 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:12.498090 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:12.499043 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:12.499111 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:12.503134 (+ 4023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:12.509223 (+ 6089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:12.509284 (+ 61us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:12.509329 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:12.510541 (+ 1212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:12.511433 (+ 892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:12.511476 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:12.511516 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:12.511970 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:12.513414 (+ 1444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:12.513473 (+ 59us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:12.514402 (+ 929us) client_negotiation.cc:769] Sending connection context
0812 03:01:12.514795 (+ 393us) client_negotiation.cc:240] Negotiation successful
0812 03:01:12.515560 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3117,"thread_start_us":2356,"threads_started":1}
I20260812 03:01:12.572270 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:12.540159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:53157 (local address 127.10.118.126:43757)
0812 03:01:12.540848 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:12.540881 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:12.544845 (+ 3964us) server_negotiation.cc:407] Connection header received
0812 03:01:12.545028 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:12.545068 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:12.545370 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:12.545555 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:12.548189 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.549898 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.553264 (+ 3366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.554388 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.554723 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:12.559051 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:12.559087 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:12.559116 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:12.559271 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:12.565196 (+ 5925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:12.566908 (+ 1712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:12.566929 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:12.566945 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:12.567135 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:12.567810 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:12.567848 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:12.567877 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:12.568559 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:12.568816 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:12.569635 (+ 819us) server_negotiation.cc:299] Negotiation successful
0812 03:01:12.570105 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260812 03:01:12.572772 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:12.540008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43757 (local address 127.10.118.66:53157)
0812 03:01:12.544169 (+ 4161us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:12.544242 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:12.544596 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:12.545654 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:12.545686 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:12.546874 (+ 1188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:12.547864 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:12.547925 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.550258 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.550295 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:12.552931 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:12.552999 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:12.554802 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:12.554844 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:12.555389 (+ 545us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:12.556479 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:12.556605 (+ 126us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:12.558647 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:12.565566 (+ 6919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:12.565599 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:12.565630 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:12.566611 (+ 981us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:12.567331 (+ 720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:12.567358 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:12.567382 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:12.567597 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:12.568768 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:12.568796 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:12.569146 (+ 350us) client_negotiation.cc:769] Sending connection context
0812 03:01:12.569416 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 03:01:12.570001 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3602,"thread_start_us":2262,"threads_started":1}
I20260812 03:01:12.650570 22526 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:45553
I20260812 03:01:12.651162 22763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:45553 every 8 connection(s)
I20260812 03:01:12.653172 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:01:12.657826 22764 heartbeater.cc:347] Connected to a master server at 127.10.118.126:43757
I20260812 03:01:12.658175 22764 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:12.659272 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 22526
I20260812 03:01:12.659439 22764 heartbeater.cc:511] Master 127.10.118.126:43757 requested a full tablet report, sending...
I20260812 03:01:12.659816 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:01:12.661300 10713 external_mini_cluster.cc:1428] Setting key d598693426c90da9f3dc445801cc675d
I20260812 03:01:12.661938 22149 ts_manager.cc:194] Registered new tserver with Master: f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553)
I20260812 03:01:12.663632 22149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:48067
WARNING: no policy specified for oryx/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.118.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab.
Entry for principal oryx/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab.
I20260812 03:01:12.716718 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43757
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab
--principal=oryx/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:13.026656 22769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:13.027225 22769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:13.027931 22769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:13.059161 22769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:13.059540 22769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:13.060189 22769 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 03:01:13.060747 22769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:01:13.096086 22769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab
--principal=oryx/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.118.126:43757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:13.097640 22769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:13.098744 22769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:13.117605 22785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:13.666962 22764 heartbeater.cc:503] Master 127.10.118.126:43757 was elected leader, sending a full tablet report...
W20260812 03:01:13.117622 22794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:13.119596 22769 server_base.cc:1034] running on GCE node
W20260812 03:01:13.117700 22782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:14.328897 22769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:14.331770 22769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:14.333242 22769 hybrid_clock.cc:648] HybridClock initialized: now 1786503674333171 us; error 88 us; skew 500 ppm
I20260812 03:01:14.334057 22769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:14 dist-test-slave-zpvx krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503674, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:14.345556 22769 init.cc:377] Logged in from keytab as oryx/127.10.118.67@KRBTEST.COM (short username oryx)
I20260812 03:01:14.351087 22769 webserver.cc:460] Webserver started at http://127.10.118.67:42121/ using document root <none> and password file <none>
I20260812 03:01:14.351991 22769 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:14.352181 22769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:14.352671 22769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:14.358214 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "fc87127101a24af78b78da0570426657"
format_stamp: "Formatted at 2026-08-12 03:01:14 on dist-test-slave-zpvx"
server_key: "1e8d5922ea091ad03fba183f2ca35877"
server_key_iv: "72be445e863ecd61831218531a097734"
server_key_version: "encryptionkey@0"
I20260812 03:01:14.359337 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "fc87127101a24af78b78da0570426657"
format_stamp: "Formatted at 2026-08-12 03:01:14 on dist-test-slave-zpvx"
server_key: "1e8d5922ea091ad03fba183f2ca35877"
server_key_iv: "72be445e863ecd61831218531a097734"
server_key_version: "encryptionkey@0"
I20260812 03:01:14.366743 22769 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 03:01:14.372468 22808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:14.373638 22769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 03:01:14.373981 22769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "fc87127101a24af78b78da0570426657"
format_stamp: "Formatted at 2026-08-12 03:01:14 on dist-test-slave-zpvx"
server_key: "1e8d5922ea091ad03fba183f2ca35877"
server_key_iv: "72be445e863ecd61831218531a097734"
server_key_version: "encryptionkey@0"
I20260812 03:01:14.374356 22769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:14.427753 22769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:14.433867 22769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:14.434381 22769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:14.446581 22769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:14.446883 22769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:14.447230 22769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:14.447480 22769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 03:01:14 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503674, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM
I20260812 03:01:14.485940 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.442031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:46793 (local address 127.10.118.126:43757)
0812 03:01:14.444433 (+ 2402us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:14.444469 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:14.447075 (+ 2606us) server_negotiation.cc:407] Connection header received
0812 03:01:14.453478 (+ 6403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:14.453517 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:14.453814 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:14.454060 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:14.456656 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.457766 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.461386 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.462625 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.462959 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.471403 (+ 8444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:14.471431 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:14.471449 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:14.471567 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:14.476314 (+ 4747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:14.477932 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:14.477974 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:14.478014 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:14.478378 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:14.479660 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:14.479701 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:14.479737 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:14.480721 (+ 984us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:14.481047 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:14.482775 (+ 1728us) server_negotiation.cc:299] Negotiation successful
0812 03:01:14.483306 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1719,"thread_start_us":1354,"threads_started":1}
I20260812 03:01:14.486832 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.442317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43757 (local address 127.10.118.67:46793)
0812 03:01:14.446759 (+ 4442us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:14.446836 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:14.453050 (+ 6214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:14.454311 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:14.454337 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:14.455257 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:14.456262 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.456313 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.458058 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.458107 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.461077 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.461123 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.463072 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.463126 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.463896 (+ 770us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.465573 (+ 1677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:14.465692 (+ 119us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:14.470963 (+ 5271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:14.476607 (+ 5644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:14.476642 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:14.476684 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:14.477587 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:14.478670 (+ 1083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:14.478719 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:14.478760 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:14.479347 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:14.481048 (+ 1701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:14.481137 (+ 89us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:14.481919 (+ 782us) client_negotiation.cc:769] Sending connection context
0812 03:01:14.482501 (+ 582us) client_negotiation.cc:240] Negotiation successful
0812 03:01:14.483348 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3935,"thread_start_us":2846,"threads_started":1}
I20260812 03:01:14.543314 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.504763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:39153 (local address 127.10.118.126:43757)
0812 03:01:14.505439 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:14.505489 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:14.509640 (+ 4151us) server_negotiation.cc:407] Connection header received
0812 03:01:14.510176 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:14.510222 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:14.510512 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:14.510768 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:14.514022 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.515866 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.519500 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.520796 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.521175 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.526750 (+ 5575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:14.526787 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:14.526816 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:14.526998 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:14.533792 (+ 6794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:14.535782 (+ 1990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:14.535819 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:14.535847 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:14.536173 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:14.537234 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:14.537282 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:14.537321 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:14.538176 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:14.538542 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:14.539862 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0812 03:01:14.540592 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260812 03:01:14.544596 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.505226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43757 (local address 127.10.118.67:39153)
0812 03:01:14.509357 (+ 4131us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:14.509423 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:14.509862 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:14.510963 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:14.511015 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:14.512588 (+ 1573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:14.513622 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.513693 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.516191 (+ 2498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.516230 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.519122 (+ 2892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.519192 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.521886 (+ 2694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.521938 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.522651 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.523880 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:14.523983 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:14.526277 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:14.534222 (+ 7945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:14.534269 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:14.534307 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:14.535444 (+ 1137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:14.536456 (+ 1012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:14.536501 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:14.536543 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:14.536919 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:14.538507 (+ 1588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:14.538548 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:14.539108 (+ 560us) client_negotiation.cc:769] Sending connection context
0812 03:01:14.539514 (+ 406us) client_negotiation.cc:240] Negotiation successful
0812 03:01:14.540301 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3499,"thread_start_us":1935,"threads_started":1}
I20260812 03:01:14.620609 22769 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:42869
I20260812 03:01:14.621261 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:42869 every 8 connection(s)
I20260812 03:01:14.623770 22769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:01:14.627182 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 22769
I20260812 03:01:14.628299 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:01:14.629101 23021 heartbeater.cc:347] Connected to a master server at 127.10.118.126:43757
I20260812 03:01:14.629580 23021 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:14.630107 10713 external_mini_cluster.cc:1428] Setting key 34a77308c02330fa159032150689725d
I20260812 03:01:14.630671 23021 heartbeater.cc:511] Master 127.10.118.126:43757 requested a full tablet report, sending...
I20260812 03:01:14.633582 22149 ts_manager.cc:194] Registered new tserver with Master: fc87127101a24af78b78da0570426657 (127.10.118.67:42869)
I20260812 03:01:14.635437 22149 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:46793
I20260812 03:01:14.636246 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 03:01:14 dist-test-slave-zpvx krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 03:01:14 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:14 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:14.716418 22822 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:14.697157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.10.118.126:43757)
0812 03:01:14.697720 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:14.697753 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:14.698856 (+ 1103us) server_negotiation.cc:407] Connection header received
0812 03:01:14.699264 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:14.699283 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:14.699466 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:14.699608 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:14.702333 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.703297 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.705328 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.705938 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.706225 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.715157 (+ 8932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52490: BlockingRecv error: recv got EOF from 127.0.0.1:52490 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 03:01:14 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503674, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM
I20260812 03:01:14.747087 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.723367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.10.118.126:43757)
0812 03:01:14.723750 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:14.723768 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:14.724700 (+ 932us) server_negotiation.cc:407] Connection header received
0812 03:01:14.724997 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:14.725017 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:14.725213 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:14.725364 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:14.727241 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.728272 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.730194 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.730849 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.731072 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.736561 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:14.736582 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:14.736601 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:14.736703 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:14.740277 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:14.741884 (+ 1607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:14.741916 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:14.741955 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:14.742246 (+ 291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:14.743032 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:14.743067 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:14.743083 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:14.743671 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:14.744148 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:14.744923 (+ 775us) server_negotiation.cc:299] Negotiation successful
0812 03:01:14.745475 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 03:01:14.754520 22149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:01:14.775300 22149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:01:14.832522 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.804577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.10.118.66:45553)
0812 03:01:14.807095 (+ 2518us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:14.807130 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:14.815050 (+ 7920us) server_negotiation.cc:407] Connection header received
0812 03:01:14.816965 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:14.817014 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:14.817484 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:14.817733 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:01:14.819849 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.822463 (+ 2614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.826742 (+ 4279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.828771 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.829100 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.829560 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:14.830586 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0812 03:01:14.830878 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1960,"thread_start_us":1636,"threads_started":1}
I20260812 03:01:14.832862 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.804141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:45553 (local address 127.0.0.1:42494)
0812 03:01:14.814698 (+ 10557us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:14.814821 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:14.815953 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:14.818443 (+ 2490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:14.818472 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:14.818577 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:01:14.819396 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.819463 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.822943 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.822995 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.826456 (+ 3461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.826497 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.829775 (+ 3278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.829797 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.830187 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.830305 (+ 118us) client_negotiation.cc:769] Sending connection context
0812 03:01:14.830510 (+ 205us) client_negotiation.cc:240] Negotiation successful
0812 03:01:14.830880 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3613,"mutex_wait_us":47,"thread_start_us":3296,"threads_started":1}
I20260812 03:01:14.837643 22652 tablet_service.cc:1467] Processing CreateTablet for tablet 93cf8ad0edd84c3fb9f88147cb430f4a (DEFAULT_TABLE table=test-table [id=2c762bfc52944558936c5b97e7344810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:14.839761 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93cf8ad0edd84c3fb9f88147cb430f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:14.840237 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.806136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.10.118.65:42009)
0812 03:01:14.826015 (+ 19879us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:14.826049 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:14.826109 (+ 60us) server_negotiation.cc:407] Connection header received
0812 03:01:14.826314 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:14.826356 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:14.826755 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:14.826998 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:01:14.828937 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.830986 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.834798 (+ 3812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.836832 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.837118 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.837538 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:14.838273 (+ 735us) server_negotiation.cc:299] Negotiation successful
0812 03:01:14.838578 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19385,"thread_start_us":19054,"threads_started":1}
I20260812 03:01:14.840456 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.806052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:42009 (local address 127.0.0.1:57308)
0812 03:01:14.824998 (+ 18946us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:14.825260 (+ 262us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:14.825497 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:14.827525 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:14.827553 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:14.827654 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:01:14.828610 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.828666 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.831328 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.831362 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.834434 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.834488 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.837340 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.837368 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.837805 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.837960 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 03:01:14.838122 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 03:01:14.838438 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12377,"mutex_wait_us":84,"thread_start_us":19707,"threads_started":1}
I20260812 03:01:14.843698 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.808503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42869 (local address 127.0.0.1:59546)
0812 03:01:14.826504 (+ 18001us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:14.826564 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:14.826956 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:14.828724 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:14.828754 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:14.828869 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:01:14.829935 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.829991 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.832906 (+ 2915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.832948 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.835895 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:14.835950 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.839681 (+ 3731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.839723 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:14.840174 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.840344 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 03:01:14.840535 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 03:01:14.840844 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17549,"mutex_wait_us":114,"thread_start_us":9534,"threads_started":1}
I20260812 03:01:14.844741 22407 tablet_service.cc:1467] Processing CreateTablet for tablet 93cf8ad0edd84c3fb9f88147cb430f4a (DEFAULT_TABLE table=test-table [id=2c762bfc52944558936c5b97e7344810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:14.845829 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:14.808886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.10.118.67:42869)
0812 03:01:14.824787 (+ 15901us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:14.824833 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:14.826700 (+ 1867us) server_negotiation.cc:407] Connection header received
0812 03:01:14.827303 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:14.827335 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:14.827667 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:14.827850 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:01:14.830312 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.832532 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.836256 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:14.838576 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:14.838921 (+ 345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:14.839415 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:14.841405 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0812 03:01:14.841716 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15158,"thread_start_us":14628,"threads_started":1}
I20260812 03:01:14.846506 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93cf8ad0edd84c3fb9f88147cb430f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:14.850402 22905 tablet_service.cc:1467] Processing CreateTablet for tablet 93cf8ad0edd84c3fb9f88147cb430f4a (DEFAULT_TABLE table=test-table [id=2c762bfc52944558936c5b97e7344810]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:14.852427 22905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93cf8ad0edd84c3fb9f88147cb430f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:14.866547 23072 tablet_bootstrap.cc:492] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Bootstrap starting.
I20260812 03:01:14.873756 23072 tablet_bootstrap.cc:654] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:14.876050 23073 tablet_bootstrap.cc:492] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Bootstrap starting.
I20260812 03:01:14.876143 23072 log.cc:824] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Log is configured to *not* fsync() on all Append() calls
I20260812 03:01:14.882614 23072 tablet_bootstrap.cc:492] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: No bootstrap required, opened a new log
I20260812 03:01:14.883262 23072 ts_tablet_manager.cc:1397] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.007s
I20260812 03:01:14.883898 23073 tablet_bootstrap.cc:654] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:14.884802 23082 tablet_bootstrap.cc:492] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Bootstrap starting.
I20260812 03:01:14.886394 23073 log.cc:824] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Log is configured to *not* fsync() on all Append() calls
I20260812 03:01:14.891956 23082 tablet_bootstrap.cc:654] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:14.892460 23073 tablet_bootstrap.cc:492] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: No bootstrap required, opened a new log
I20260812 03:01:14.892982 23073 ts_tablet_manager.cc:1397] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.007s
I20260812 03:01:14.894340 23082 log.cc:824] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Log is configured to *not* fsync() on all Append() calls
I20260812 03:01:14.901053 23082 tablet_bootstrap.cc:492] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: No bootstrap required, opened a new log
I20260812 03:01:14.901557 23082 ts_tablet_manager.cc:1397] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260812 03:01:14.907805 23072 raft_consensus.cc:348] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:14.908494 23072 raft_consensus.cc:374] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:14.908782 23072 raft_consensus.cc:729] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04caa54d700475697a9dcfe72b8d7b4, State: Initialized, Role: FOLLOWER
I20260812 03:01:14.909614 23072 consensus_queue.cc:260] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:14.913321 23072 ts_tablet_manager.cc:1428] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Time spent starting tablet: real 0.030s user 0.019s sys 0.010s
I20260812 03:01:14.923699 23073 raft_consensus.cc:348] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:14.924702 23073 raft_consensus.cc:374] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:14.925019 23073 raft_consensus.cc:729] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09f723a9a2408c8a3c98a88bb8ca56, State: Initialized, Role: FOLLOWER
I20260812 03:01:14.925933 23073 consensus_queue.cc:260] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:14.929695 23073 ts_tablet_manager.cc:1428] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent starting tablet: real 0.036s user 0.030s sys 0.005s
I20260812 03:01:14.932806 23082 raft_consensus.cc:348] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:14.933521 23082 raft_consensus.cc:374] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:14.933837 23082 raft_consensus.cc:729] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc87127101a24af78b78da0570426657, State: Initialized, Role: FOLLOWER
I20260812 03:01:14.934770 23082 consensus_queue.cc:260] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:14.937747 23021 heartbeater.cc:503] Master 127.10.118.126:43757 was elected leader, sending a full tablet report...
I20260812 03:01:14.938321 23082 ts_tablet_manager.cc:1428] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Time spent starting tablet: real 0.036s user 0.027s sys 0.008s
W20260812 03:01:15.005980 22521 tablet.cc:2369] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 03:01:15.129151 23023 tablet.cc:2369] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 03:01:15.160022 22766 tablet.cc:2369] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:01:15.291852 23092 raft_consensus.cc:482] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:01:15.292441 23092 raft_consensus.cc:504] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.294783 23092 leader_election.cc:288] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553), 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009)
I20260812 03:01:15.307562 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:15.295391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:57263 (local address 127.10.118.66:45553)
0812 03:01:15.296083 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:15.296111 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:15.297901 (+ 1790us) server_negotiation.cc:407] Connection header received
0812 03:01:15.298371 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:15.298392 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:15.298649 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:15.298805 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:01:15.299739 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.300748 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.303552 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.304632 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.304990 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:15.305256 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:15.305746 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 03:01:15.306013 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260812 03:01:15.307698 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:15.295927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:55923 (local address 127.10.118.65:42009)
0812 03:01:15.296788 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:15.296827 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:15.297942 (+ 1115us) server_negotiation.cc:407] Connection header received
0812 03:01:15.298399 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:15.298419 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:15.298716 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:15.298903 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:01:15.300187 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.301133 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.303730 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.304786 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.305159 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:15.305422 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:15.305957 (+ 535us) server_negotiation.cc:299] Negotiation successful
0812 03:01:15.306224 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260812 03:01:15.307940 23095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:15.295196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:45553 (local address 127.10.118.67:57263)
0812 03:01:15.297723 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:15.297771 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:15.298115 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:15.298862 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:15.298890 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:15.298990 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:01:15.299469 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:15.299505 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.300972 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.300999 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:15.303280 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:15.303319 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.304917 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.304954 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:15.305368 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:15.305492 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 03:01:15.305654 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 03:01:15.305940 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2018,"mutex_wait_us":225,"spinlock_wait_cycles":45696,"thread_start_us":1698,"threads_started":1}
I20260812 03:01:15.309876 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93cf8ad0edd84c3fb9f88147cb430f4a" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4" is_pre_election: true
I20260812 03:01:15.310755 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:15.295347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:42009 (local address 127.10.118.67:55923)
0812 03:01:15.297737 (+ 2390us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:15.297783 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:15.298133 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:15.299061 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:15.299089 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:15.299190 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:01:15.299872 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:15.299930 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.301375 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.301420 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:15.303457 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:15.303494 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.305104 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.305136 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:15.305596 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:15.305722 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 03:01:15.305900 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 03:01:15.308885 (+ 2985us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1994,"mutex_wait_us":238,"spinlock_wait_cycles":66688,"thread_start_us":1930,"threads_started":1}
I20260812 03:01:15.310837 22689 raft_consensus.cc:2454] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:15.312206 22809 leader_election.cc:302] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f04caa54d700475697a9dcfe72b8d7b4, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:15.312664 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93cf8ad0edd84c3fb9f88147cb430f4a" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" is_pre_election: true
I20260812 03:01:15.313320 23092 raft_consensus.cc:2781] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:01:15.313656 23092 raft_consensus.cc:482] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:01:15.313556 22446 raft_consensus.cc:2454] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:15.313969 23092 raft_consensus.cc:3037] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:15.319113 23092 raft_consensus.cc:504] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.321007 23092 leader_election.cc:288] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Requested vote from peers f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553), 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009)
I20260812 03:01:15.321447 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93cf8ad0edd84c3fb9f88147cb430f4a" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4"
I20260812 03:01:15.322011 22689 raft_consensus.cc:3037] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:15.321879 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93cf8ad0edd84c3fb9f88147cb430f4a" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56"
I20260812 03:01:15.322391 22446 raft_consensus.cc:3037] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:15.326987 22689 raft_consensus.cc:2454] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:15.327845 22809 leader_election.cc:302] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f04caa54d700475697a9dcfe72b8d7b4, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:15.328553 23092 raft_consensus.cc:2781] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:15.329272 22446 raft_consensus.cc:2454] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:15.330271 23092 raft_consensus.cc:686] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 1 LEADER]: Becoming Leader. State: Replica: fc87127101a24af78b78da0570426657, State: Running, Role: LEADER
I20260812 03:01:15.331194 23092 consensus_queue.cc:237] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.344460 22150 catalog_manager.cc:5495] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc87127101a24af78b78da0570426657 (127.10.118.67). New cstate: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } health_report { overall_health: HEALTHY } } }
I20260812 03:01:15.418669 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:15.407938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.10.118.67:42869)
0812 03:01:15.409981 (+ 2043us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:15.410002 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:15.410040 (+ 38us) server_negotiation.cc:407] Connection header received
0812 03:01:15.410190 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:15.410206 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:15.410431 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:15.410585 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:01:15.411334 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.412506 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.414390 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.414959 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.415246 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:15.416102 (+ 856us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:01:15.416796 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:01:15.416989 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:15.417295 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 03:01:15.417503 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1746,"thread_start_us":1465,"threads_started":1}
I20260812 03:01:15.431361 22689 raft_consensus.cc:1261] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Refusing update from remote peer fc87127101a24af78b78da0570426657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:15.431493 22446 raft_consensus.cc:1261] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Refusing update from remote peer fc87127101a24af78b78da0570426657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:15.433055 23098 consensus_queue.cc:1035] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 03:01:15.433763 23092 consensus_queue.cc:1035] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:01:15.490410 22149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:15.491103 22149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:15.499575 22149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Sending DeleteTablet for 3 replicas of tablet 93cf8ad0edd84c3fb9f88147cb430f4a
I20260812 03:01:15.501693 22652 tablet_service.cc:1514] Processing DeleteTablet for tablet 93cf8ad0edd84c3fb9f88147cb430f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:15 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:42494
I20260812 03:01:15.502663 22905 tablet_service.cc:1514] Processing DeleteTablet for tablet 93cf8ad0edd84c3fb9f88147cb430f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:15 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:59546
I20260812 03:01:15.503268 22407 tablet_service.cc:1514] Processing DeleteTablet for tablet 93cf8ad0edd84c3fb9f88147cb430f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:15 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57308
I20260812 03:01:15.511605 22149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 03:01:15.512271 23133 tablet_replica.cc:330] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: stopping tablet replica
I20260812 03:01:15.513325 23133 raft_consensus.cc:2229] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260812 03:01:15.513921 22149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:01:15.516914 23133 raft_consensus.cc:2258] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:15.520545 23134 tablet_replica.cc:330] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: stopping tablet replica
I20260812 03:01:15.521641 23134 raft_consensus.cc:2229] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:01:15.522948 23134 raft_consensus.cc:2258] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:15.523247 23132 tablet_replica.cc:330] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: stopping tablet replica
I20260812 03:01:15.524257 23133 ts_tablet_manager.cc:1905] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:15.526141 23134 ts_tablet_manager.cc:1905] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:15.528275 23132 raft_consensus.cc:2229] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:01:15.529263 23132 raft_consensus.cc:2258] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:15.537932 22407 tablet_service.cc:1467] Processing CreateTablet for tablet 17c9321221924b97b8db60c4ef4d58f2 (DEFAULT_TABLE table=test-table [id=05dcba4b8c5245cd8440f57ef4d3e4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:15.539052 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c9321221924b97b8db60c4ef4d58f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:15.542234 23134 ts_tablet_manager.cc:1918] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:01:15.542768 23134 log.cc:1196] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/93cf8ad0edd84c3fb9f88147cb430f4a
I20260812 03:01:15.542732 23132 ts_tablet_manager.cc:1905] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:15.543720 23134 ts_tablet_manager.cc:1939] T 93cf8ad0edd84c3fb9f88147cb430f4a P fc87127101a24af78b78da0570426657: Deleting consensus metadata
I20260812 03:01:15.546988 22124 catalog_manager.cc:4841] TS fc87127101a24af78b78da0570426657 (127.10.118.67:42869): tablet 93cf8ad0edd84c3fb9f88147cb430f4a (table test-table [id=2c762bfc52944558936c5b97e7344810]) successfully deleted
I20260812 03:01:15.550084 22652 tablet_service.cc:1467] Processing CreateTablet for tablet 17c9321221924b97b8db60c4ef4d58f2 (DEFAULT_TABLE table=test-table [id=05dcba4b8c5245cd8440f57ef4d3e4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:15.550859 22905 tablet_service.cc:1467] Processing CreateTablet for tablet 17c9321221924b97b8db60c4ef4d58f2 (DEFAULT_TABLE table=test-table [id=05dcba4b8c5245cd8440f57ef4d3e4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:15.551177 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c9321221924b97b8db60c4ef4d58f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:15.551896 22905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c9321221924b97b8db60c4ef4d58f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:15.557389 23132 ts_tablet_manager.cc:1918] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:01:15.557904 23132 log.cc:1196] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/93cf8ad0edd84c3fb9f88147cb430f4a
I20260812 03:01:15.558746 23132 ts_tablet_manager.cc:1939] T 93cf8ad0edd84c3fb9f88147cb430f4a P f04caa54d700475697a9dcfe72b8d7b4: Deleting consensus metadata
I20260812 03:01:15.561918 22122 catalog_manager.cc:4841] TS f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553): tablet 93cf8ad0edd84c3fb9f88147cb430f4a (table test-table [id=2c762bfc52944558936c5b97e7344810]) successfully deleted
I20260812 03:01:15.564061 23133 ts_tablet_manager.cc:1918] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:01:15.564759 23133 log.cc:1196] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/93cf8ad0edd84c3fb9f88147cb430f4a
I20260812 03:01:15.565677 23133 ts_tablet_manager.cc:1939] T 93cf8ad0edd84c3fb9f88147cb430f4a P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting consensus metadata
I20260812 03:01:15.569711 22125 catalog_manager.cc:4841] TS 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009): tablet 93cf8ad0edd84c3fb9f88147cb430f4a (table test-table [id=2c762bfc52944558936c5b97e7344810]) successfully deleted
I20260812 03:01:15.571858 23135 tablet_bootstrap.cc:492] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: Bootstrap starting.
I20260812 03:01:15.577754 23135 tablet_bootstrap.cc:654] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:15.589588 23137 tablet_bootstrap.cc:492] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: Bootstrap starting.
I20260812 03:01:15.594218 23140 tablet_bootstrap.cc:492] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: Bootstrap starting.
I20260812 03:01:15.595542 23135 tablet_bootstrap.cc:492] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: No bootstrap required, opened a new log
I20260812 03:01:15.595953 23135 ts_tablet_manager.cc:1397] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.015s
I20260812 03:01:15.596110 23137 tablet_bootstrap.cc:654] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:15.599831 23140 tablet_bootstrap.cc:654] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:15.600306 23135 raft_consensus.cc:348] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.601405 23135 raft_consensus.cc:374] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:15.601837 23135 raft_consensus.cc:729] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04caa54d700475697a9dcfe72b8d7b4, State: Initialized, Role: FOLLOWER
I20260812 03:01:15.602875 23135 consensus_queue.cc:260] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.607015 23135 ts_tablet_manager.cc:1428] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260812 03:01:15.611541 23137 tablet_bootstrap.cc:492] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: No bootstrap required, opened a new log
I20260812 03:01:15.612002 23137 ts_tablet_manager.cc:1397] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.010s
I20260812 03:01:15.612334 23140 tablet_bootstrap.cc:492] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: No bootstrap required, opened a new log
I20260812 03:01:15.612880 23140 ts_tablet_manager.cc:1397] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.014s
I20260812 03:01:15.614596 23137 raft_consensus.cc:348] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.615412 23137 raft_consensus.cc:374] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:15.615738 23137 raft_consensus.cc:729] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc87127101a24af78b78da0570426657, State: Initialized, Role: FOLLOWER
I20260812 03:01:15.615486 23140 raft_consensus.cc:348] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.616400 23140 raft_consensus.cc:374] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:15.616745 23140 raft_consensus.cc:729] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09f723a9a2408c8a3c98a88bb8ca56, State: Initialized, Role: FOLLOWER
I20260812 03:01:15.617518 23140 consensus_queue.cc:260] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.616528 23137 consensus_queue.cc:260] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.619846 23140 ts_tablet_manager.cc:1428] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260812 03:01:15.622262 23137 ts_tablet_manager.cc:1428] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260812 03:01:15.634706 23098 raft_consensus.cc:482] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:01:15.635067 23098 raft_consensus.cc:504] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.636754 23098 leader_election.cc:288] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009), f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553)
I20260812 03:01:15.637509 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c9321221924b97b8db60c4ef4d58f2" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" is_pre_election: true
I20260812 03:01:15.637866 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c9321221924b97b8db60c4ef4d58f2" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4" is_pre_election: true
I20260812 03:01:15.638123 22446 raft_consensus.cc:2454] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:15.638495 22689 raft_consensus.cc:2454] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:15.639556 22813 leader_election.cc:302] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09f723a9a2408c8a3c98a88bb8ca56, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:15.640422 23098 raft_consensus.cc:2781] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:01:15.640733 23098 raft_consensus.cc:482] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:01:15.641011 23098 raft_consensus.cc:3037] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:15.645593 23098 raft_consensus.cc:504] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.647392 23098 leader_election.cc:288] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Requested vote from peers 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009), f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553)
I20260812 03:01:15.647921 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c9321221924b97b8db60c4ef4d58f2" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56"
I20260812 03:01:15.648222 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17c9321221924b97b8db60c4ef4d58f2" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4"
I20260812 03:01:15.648500 22446 raft_consensus.cc:3037] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:15.648795 22689 raft_consensus.cc:3037] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:15.655181 22689 raft_consensus.cc:2454] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:15.655189 22446 raft_consensus.cc:2454] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:15.656303 22813 leader_election.cc:302] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09f723a9a2408c8a3c98a88bb8ca56, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:15.657369 23098 raft_consensus.cc:2781] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:15.658576 23098 raft_consensus.cc:686] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 1 LEADER]: Becoming Leader. State: Replica: fc87127101a24af78b78da0570426657, State: Running, Role: LEADER
I20260812 03:01:15.659412 23098 consensus_queue.cc:237] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.685298 22150 catalog_manager.cc:5495] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc87127101a24af78b78da0570426657 (127.10.118.67). New cstate: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } health_report { overall_health: HEALTHY } } }
Aug 12 03:01:15 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503668, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.126@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM
I20260812 03:01:15.797331 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:15.742508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43757 (local address 127.0.0.1:55446)
0812 03:01:15.766983 (+ 24475us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:15.767039 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:15.767373 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:15.768209 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:15.768241 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:15.769600 (+ 1359us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:15.770326 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:15.770379 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.772444 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.772479 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:15.774718 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:15.774773 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.776312 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.776348 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:15.776847 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:15.777963 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:15.778060 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:15.782327 (+ 4267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:15.789153 (+ 6826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:15.789205 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:15.789236 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:15.790283 (+ 1047us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:15.791132 (+ 849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:15.791161 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:15.791182 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:15.791478 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:15.792878 (+ 1400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:15.792925 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:15.793351 (+ 426us) client_negotiation.cc:769] Sending connection context
0812 03:01:15.793688 (+ 337us) client_negotiation.cc:240] Negotiation successful
0812 03:01:15.794360 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23946,"thread_start_us":23591,"threads_started":1}
I20260812 03:01:15.798497 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:15.744113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.10.118.126:43757)
0812 03:01:15.765881 (+ 21768us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:15.765908 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:15.767218 (+ 1310us) server_negotiation.cc:407] Connection header received
0812 03:01:15.767645 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:15.767672 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:15.767887 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:15.768074 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:15.770679 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.772145 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.775016 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:15.775994 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:15.776265 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:15.782740 (+ 6475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:15.782780 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:15.782810 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:15.782959 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:15.788902 (+ 5943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:15.790527 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:15.790561 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:15.790587 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:15.790904 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:15.791703 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:15.791737 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:15.791762 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:15.792625 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:15.792906 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:15.794039 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0812 03:01:15.794770 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":21265,"thread_start_us":20959,"threads_started":1}
I20260812 03:01:15.808753 22150 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:55446:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 03:01:15.811471 22150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:01:15.831687 22407 tablet_service.cc:1467] Processing CreateTablet for tablet f2afd42cc24040ee83297181ff996934 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5097028bd78245b38606395e0c83cc0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 03:01:15.833154 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd42cc24040ee83297181ff996934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:15.832839 22652 tablet_service.cc:1467] Processing CreateTablet for tablet f2afd42cc24040ee83297181ff996934 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5097028bd78245b38606395e0c83cc0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 03:01:15.832829 22905 tablet_service.cc:1467] Processing CreateTablet for tablet f2afd42cc24040ee83297181ff996934 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5097028bd78245b38606395e0c83cc0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 03:01:15.834072 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd42cc24040ee83297181ff996934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:15.834072 22905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2afd42cc24040ee83297181ff996934. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:15.849753 23137 tablet_bootstrap.cc:492] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: Bootstrap starting.
I20260812 03:01:15.851398 23135 tablet_bootstrap.cc:492] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: Bootstrap starting.
I20260812 03:01:15.855813 23137 tablet_bootstrap.cc:654] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:15.857373 23135 tablet_bootstrap.cc:654] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:15.858129 23140 tablet_bootstrap.cc:492] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: Bootstrap starting.
I20260812 03:01:15.865504 23140 tablet_bootstrap.cc:654] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:15.872756 23135 tablet_bootstrap.cc:492] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: No bootstrap required, opened a new log
I20260812 03:01:15.873204 23135 ts_tablet_manager.cc:1397] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.007s
I20260812 03:01:15.873560 23140 tablet_bootstrap.cc:492] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: No bootstrap required, opened a new log
I20260812 03:01:15.873965 23140 ts_tablet_manager.cc:1397] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260812 03:01:15.874634 23137 tablet_bootstrap.cc:492] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: No bootstrap required, opened a new log
I20260812 03:01:15.875034 23137 ts_tablet_manager.cc:1397] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.005s
I20260812 03:01:15.875911 23135 raft_consensus.cc:348] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.876726 23135 raft_consensus.cc:374] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:15.876538 23140 raft_consensus.cc:348] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.877022 23135 raft_consensus.cc:729] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04caa54d700475697a9dcfe72b8d7b4, State: Initialized, Role: FOLLOWER
I20260812 03:01:15.877254 23140 raft_consensus.cc:374] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:15.877527 23140 raft_consensus.cc:729] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09f723a9a2408c8a3c98a88bb8ca56, State: Initialized, Role: FOLLOWER
I20260812 03:01:15.877575 23137 raft_consensus.cc:348] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.878557 23137 raft_consensus.cc:374] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:15.878870 23137 raft_consensus.cc:729] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc87127101a24af78b78da0570426657, State: Initialized, Role: FOLLOWER
I20260812 03:01:15.878445 23140 consensus_queue.cc:260] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.879686 23137 consensus_queue.cc:260] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.879988 23117 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:15.880803 23117 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 03:01:15.880337 23135 consensus_queue.cc:260] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:15.880942 23098 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:15.881626 23098 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 03:01:15.883168 23135 ts_tablet_manager.cc:1428] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20260812 03:01:15.883363 23119 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:15.884204 23119 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 03:01:15.884569 23137 ts_tablet_manager.cc:1428] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260812 03:01:15.892915 23140 ts_tablet_manager.cc:1428] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260812 03:01:16.132697 23098 raft_consensus.cc:482] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:01:16.133121 23098 raft_consensus.cc:504] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.134708 23098 leader_election.cc:288] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009), f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553)
I20260812 03:01:16.135579 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd42cc24040ee83297181ff996934" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" is_pre_election: true
I20260812 03:01:16.135883 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd42cc24040ee83297181ff996934" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4" is_pre_election: true
I20260812 03:01:16.136243 22446 raft_consensus.cc:2454] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:16.136526 22689 raft_consensus.cc:2454] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:16.137259 22813 leader_election.cc:302] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09f723a9a2408c8a3c98a88bb8ca56, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:16.138051 23098 raft_consensus.cc:2781] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:01:16.138404 23098 raft_consensus.cc:482] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:01:16.138721 23098 raft_consensus.cc:3037] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:16.144991 23098 raft_consensus.cc:504] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.146777 23098 leader_election.cc:288] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Requested vote from peers 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009), f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553)
I20260812 03:01:16.147524 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd42cc24040ee83297181ff996934" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56"
I20260812 03:01:16.148057 22446 raft_consensus.cc:3037] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:16.147949 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2afd42cc24040ee83297181ff996934" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4"
I20260812 03:01:16.148535 22689 raft_consensus.cc:3037] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:16.152478 22446 raft_consensus.cc:2454] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:16.153335 22813 leader_election.cc:302] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09f723a9a2408c8a3c98a88bb8ca56, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:16.153975 23098 raft_consensus.cc:2781] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:16.154383 23098 raft_consensus.cc:686] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [term 1 LEADER]: Becoming Leader. State: Replica: fc87127101a24af78b78da0570426657, State: Running, Role: LEADER
I20260812 03:01:16.154626 22689 raft_consensus.cc:2454] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:16.155133 23098 consensus_queue.cc:237] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.160838 23092 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: TxnStatusTablet state changed. Reason: New leader fc87127101a24af78b78da0570426657. Latest consensus state: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:16.161522 23092 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: This TxnStatusTablet replica's current role is: LEADER
I20260812 03:01:16.162633 23172 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 03:01:16.163120 23172 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 03:01:16.166453 22149 catalog_manager.cc:5495] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc87127101a24af78b78da0570426657 (127.10.118.67). New cstate: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } health_report { overall_health: HEALTHY } } }
I20260812 03:01:16.191346 23092 consensus_queue.cc:1035] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:01:16.203298 23092 consensus_queue.cc:1035] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:01:16.262274 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.251464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42869 (local address 127.0.0.1:36612)
0812 03:01:16.252068 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:16.252111 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:16.252426 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:16.253940 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:16.253996 (+ 56us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:16.254080 (+ 84us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 03:01:16.254552 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.254599 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.256661 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.256684 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.258003 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.258040 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.259445 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.259465 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.259726 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.259855 (+ 129us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 03:01:16.260688 (+ 833us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 03:01:16.260738 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 03:01:16.260877 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 03:01:16.261185 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 03:01:16.262840 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.251692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.10.118.67:42869)
0812 03:01:16.252965 (+ 1273us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:16.252980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:16.253046 (+ 66us) server_negotiation.cc:407] Connection header received
0812 03:01:16.253222 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:16.253253 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:16.253555 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:16.253797 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:01:16.254905 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.256439 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.258267 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.259206 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.259488 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.260159 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:01:16.260487 (+ 328us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:01:16.260622 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:16.260968 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 03:01:16.261148 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":642,"threads_started":1}
I20260812 03:01:16.269266 22689 raft_consensus.cc:1261] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Refusing update from remote peer fc87127101a24af78b78da0570426657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:16.269443 22446 raft_consensus.cc:1261] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Refusing update from remote peer fc87127101a24af78b78da0570426657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:16.270480 23098 consensus_queue.cc:1035] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:01:16.271108 23092 consensus_queue.cc:1035] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:01:16.285385 23117 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: TxnStatusTablet state changed. Reason: New leader fc87127101a24af78b78da0570426657. Latest consensus state: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:16.286029 23117 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 03:01:16.290258 23119 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: TxnStatusTablet state changed. Reason: New leader fc87127101a24af78b78da0570426657. Latest consensus state: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:16.290944 23119 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 03:01:16.296600 23175 mvcc.cc:205] Tried to move back new op lower bound from 7317519057986904064 to 7317519057540546560. Current Snapshot: MvccSnapshot[applied={T|T < 7317519057986904064}]
I20260812 03:01:16.306125 23098 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:16.306813 23098 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: This TxnStatusTablet replica's current role is: LEADER
I20260812 03:01:16.308643 23119 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:16.309295 23119 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P f04caa54d700475697a9dcfe72b8d7b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 03:01:16.321540 23179 mvcc.cc:205] Tried to move back new op lower bound from 7317519057986904064 to 7317519057540546560. Current Snapshot: MvccSnapshot[applied={T|T < 7317519057986904064}]
I20260812 03:01:16.325893 23117 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:16.326705 23117 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P 8d09f723a9a2408c8a3c98a88bb8ca56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 03:01:16.335534 23092 tablet_replica.cc:439] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } } }
I20260812 03:01:16.336238 23092 tablet_replica.cc:442] T f2afd42cc24040ee83297181ff996934 P fc87127101a24af78b78da0570426657: This TxnStatusTablet replica's current role is: LEADER
I20260812 03:01:16.372289 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.359421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:58135 (local address 127.10.118.67:42869)
0812 03:01:16.359926 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:16.359952 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:16.363431 (+ 3479us) server_negotiation.cc:407] Connection header received
0812 03:01:16.363847 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:16.363866 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:16.364069 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:16.364240 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:01:16.365187 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.366038 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.367933 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.368580 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.368854 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.369712 (+ 858us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:01:16.369907 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:01:16.370051 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:16.370324 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 03:01:16.370527 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":83}
I20260812 03:01:16.373396 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.359339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42869 (local address 127.10.118.67:58135)
0812 03:01:16.363276 (+ 3937us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:16.363318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:16.363616 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:16.364289 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:16.364315 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:16.364407 (+ 92us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 03:01:16.364933 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.364970 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.366262 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.366299 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.367694 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.367728 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.368854 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.368885 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.369287 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.369412 (+ 125us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 03:01:16.370084 (+ 672us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 03:01:16.370123 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 03:01:16.370264 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 03:01:16.370613 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3499,"mutex_wait_us":21,"thread_start_us":3217,"threads_started":1}
I20260812 03:01:16.520095 22148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:16.520710 22148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:16.533387 22148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Sending DeleteTablet for 3 replicas of tablet 17c9321221924b97b8db60c4ef4d58f2
I20260812 03:01:16.536654 22652 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c9321221924b97b8db60c4ef4d58f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:16 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:42494
I20260812 03:01:16.537884 22407 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c9321221924b97b8db60c4ef4d58f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:16 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57308
I20260812 03:01:16.540134 23208 tablet_replica.cc:330] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: stopping tablet replica
I20260812 03:01:16.541792 23208 raft_consensus.cc:2229] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:01:16.542465 23208 raft_consensus.cc:2258] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:16.546144 23208 ts_tablet_manager.cc:1905] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:16.548538 23207 tablet_replica.cc:330] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: stopping tablet replica
I20260812 03:01:16.549696 23207 raft_consensus.cc:2229] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:01:16.550290 23207 raft_consensus.cc:2258] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:16.553218 23207 ts_tablet_manager.cc:1905] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:16.570964 23207 ts_tablet_manager.cc:1918] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 03:01:16.571437 23207 log.cc:1196] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/17c9321221924b97b8db60c4ef4d58f2
I20260812 03:01:16.572387 23207 ts_tablet_manager.cc:1939] T 17c9321221924b97b8db60c4ef4d58f2 P f04caa54d700475697a9dcfe72b8d7b4: Deleting consensus metadata
I20260812 03:01:16.575259 22122 catalog_manager.cc:4841] TS f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553): tablet 17c9321221924b97b8db60c4ef4d58f2 (table test-table [id=05dcba4b8c5245cd8440f57ef4d3e4fe]) successfully deleted
I20260812 03:01:16.595568 23208 ts_tablet_manager.cc:1918] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 03:01:16.596007 23208 log.cc:1196] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/17c9321221924b97b8db60c4ef4d58f2
I20260812 03:01:16.596916 23208 ts_tablet_manager.cc:1939] T 17c9321221924b97b8db60c4ef4d58f2 P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting consensus metadata
I20260812 03:01:16.599742 22125 catalog_manager.cc:4841] TS 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009): tablet 17c9321221924b97b8db60c4ef4d58f2 (table test-table [id=05dcba4b8c5245cd8440f57ef4d3e4fe]) successfully deleted
I20260812 03:01:16.614807 22905 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c9321221924b97b8db60c4ef4d58f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:16 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:59546
Aug 12 03:01:16 dist-test-slave-zpvx krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260812 03:01:16.617461 23211 tablet_replica.cc:330] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: stopping tablet replica
I20260812 03:01:16.618251 23211 raft_consensus.cc:2229] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:01:16.619143 23211 raft_consensus.cc:2258] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:16.621760 23211 ts_tablet_manager.cc:1905] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:16.637979 23211 ts_tablet_manager.cc:1918] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 03:01:16.638357 23211 log.cc:1196] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/17c9321221924b97b8db60c4ef4d58f2
I20260812 03:01:16.639094 23211 ts_tablet_manager.cc:1939] T 17c9321221924b97b8db60c4ef4d58f2 P fc87127101a24af78b78da0570426657: Deleting consensus metadata
I20260812 03:01:16.641651 22124 catalog_manager.cc:4841] TS fc87127101a24af78b78da0570426657 (127.10.118.67:42869): tablet 17c9321221924b97b8db60c4ef4d58f2 (table test-table [id=05dcba4b8c5245cd8440f57ef4d3e4fe]) successfully deleted
Aug 12 03:01:16 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:16 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:16.665885 23224 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:16.637118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.10.118.126:43757)
0812 03:01:16.643428 (+ 6310us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:16.643451 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:16.644340 (+ 889us) server_negotiation.cc:407] Connection header received
0812 03:01:16.647096 (+ 2756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:16.647124 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:16.647361 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:16.647541 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:16.650672 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.652224 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.655016 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.655949 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.656229 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.664346 (+ 8117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55462: BlockingRecv error: recv got EOF from 127.0.0.1:55462 (error 108)
Metrics: {"server-negotiator.queue_time_us":5878,"thread_start_us":5619,"threads_started":1}
Aug 12 03:01:16 dist-test-slave-zpvx krb5kdc[22061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM
I20260812 03:01:16.704005 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.675952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.10.118.126:43757)
0812 03:01:16.676479 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:16.676514 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:16.677333 (+ 819us) server_negotiation.cc:407] Connection header received
0812 03:01:16.677698 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:16.677721 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:16.677887 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:16.678052 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:16.680299 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.681253 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.683025 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.683747 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.684034 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.690841 (+ 6807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:16.690873 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:16.690897 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:16.691046 (+ 149us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:16.696318 (+ 5272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:16.697908 (+ 1590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:16.697937 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:16.697958 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:16.698219 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:16.699004 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:16.699036 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:16.699058 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:16.699740 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:16.700726 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:16.701567 (+ 841us) server_negotiation.cc:299] Negotiation successful
0812 03:01:16.702024 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 03:01:16.715548 22148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:01:16.717581 22148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:01:16.738308 22407 tablet_service.cc:1467] Processing CreateTablet for tablet 44660217c0354a2a996e731e1aeb7c6d (DEFAULT_TABLE table=test-table [id=e165f979846e4753b0e51bd5225368ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:16.739401 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44660217c0354a2a996e731e1aeb7c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:16.741549 22652 tablet_service.cc:1467] Processing CreateTablet for tablet 44660217c0354a2a996e731e1aeb7c6d (DEFAULT_TABLE table=test-table [id=e165f979846e4753b0e51bd5225368ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:16.741722 22905 tablet_service.cc:1467] Processing CreateTablet for tablet 44660217c0354a2a996e731e1aeb7c6d (DEFAULT_TABLE table=test-table [id=e165f979846e4753b0e51bd5225368ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:16.742635 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44660217c0354a2a996e731e1aeb7c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:16.742777 22905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44660217c0354a2a996e731e1aeb7c6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:16.788110 23236 tablet_bootstrap.cc:492] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: Bootstrap starting.
I20260812 03:01:16.790544 23240 tablet_bootstrap.cc:492] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: Bootstrap starting.
I20260812 03:01:16.795287 23237 tablet_bootstrap.cc:492] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: Bootstrap starting.
I20260812 03:01:16.795753 23236 tablet_bootstrap.cc:654] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:16.807528 23240 tablet_bootstrap.cc:654] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:16.809558 23237 tablet_bootstrap.cc:654] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:16.811429 23236 tablet_bootstrap.cc:492] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: No bootstrap required, opened a new log
I20260812 03:01:16.811844 23236 ts_tablet_manager.cc:1397] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.010s
I20260812 03:01:16.823936 23240 tablet_bootstrap.cc:492] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: No bootstrap required, opened a new log
I20260812 03:01:16.824425 23240 ts_tablet_manager.cc:1397] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.011s
I20260812 03:01:16.827409 23240 raft_consensus.cc:348] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.828260 23240 raft_consensus.cc:374] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:16.814671 23236 raft_consensus.cc:348] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.828634 23240 raft_consensus.cc:729] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09f723a9a2408c8a3c98a88bb8ca56, State: Initialized, Role: FOLLOWER
I20260812 03:01:16.828979 23236 raft_consensus.cc:374] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:16.827764 23237 tablet_bootstrap.cc:492] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: No bootstrap required, opened a new log
I20260812 03:01:16.829308 23236 raft_consensus.cc:729] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04caa54d700475697a9dcfe72b8d7b4, State: Initialized, Role: FOLLOWER
I20260812 03:01:16.829761 23237 ts_tablet_manager.cc:1397] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.016s
I20260812 03:01:16.830116 23236 consensus_queue.cc:260] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.830433 23240 consensus_queue.cc:260] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.838332 23240 ts_tablet_manager.cc:1428] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260812 03:01:16.839315 23236 ts_tablet_manager.cc:1428] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: Time spent starting tablet: real 0.027s user 0.006s sys 0.002s
I20260812 03:01:16.842092 23237 raft_consensus.cc:348] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.842836 23237 raft_consensus.cc:374] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:16.843125 23237 raft_consensus.cc:729] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc87127101a24af78b78da0570426657, State: Initialized, Role: FOLLOWER
I20260812 03:01:16.844028 23237 consensus_queue.cc:260] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.848263 23237 ts_tablet_manager.cc:1428] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260812 03:01:16.949113 23251 raft_consensus.cc:482] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:01:16.949640 23251 raft_consensus.cc:504] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:16.952617 23251 leader_election.cc:288] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553), fc87127101a24af78b78da0570426657 (127.10.118.67:42869)
I20260812 03:01:16.985780 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.953006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:45553 (local address 127.10.118.65:46571)
0812 03:01:16.968800 (+ 15794us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:16.968877 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:16.969230 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:16.974538 (+ 5308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:16.974567 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:16.974672 (+ 105us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:01:16.975374 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.975410 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.977394 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.977426 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.980809 (+ 3383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.980861 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.982526 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.982559 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.983017 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.983167 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 03:01:16.983321 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 03:01:16.983629 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15115,"mutex_wait_us":172,"thread_start_us":20202,"threads_started":1}
I20260812 03:01:16.985889 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.955465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:44023 (local address 127.10.118.67:42869)
0812 03:01:16.973993 (+ 18528us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:16.974007 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:16.974043 (+ 36us) server_negotiation.cc:407] Connection header received
0812 03:01:16.974163 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:16.974186 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:16.974364 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:16.974558 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:01:16.976410 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.977673 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.980956 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.982705 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.983061 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.983315 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:16.984026 (+ 711us) server_negotiation.cc:299] Negotiation successful
0812 03:01:16.984316 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18201,"thread_start_us":16946,"threads_started":1}
I20260812 03:01:16.987658 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.953493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:46571 (local address 127.10.118.66:45553)
0812 03:01:16.973759 (+ 20266us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:16.973783 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:16.973830 (+ 47us) server_negotiation.cc:407] Connection header received
0812 03:01:16.973981 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:16.974012 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:16.974268 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:16.974437 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:01:16.975662 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.977104 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.981101 (+ 3997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.982199 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.982579 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.982855 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:16.986456 (+ 3601us) server_negotiation.cc:299] Negotiation successful
0812 03:01:16.986721 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19827,"thread_start_us":19551,"threads_started":1}
I20260812 03:01:16.986361 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:16.953124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42869 (local address 127.10.118.65:44023)
0812 03:01:16.971468 (+ 18344us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:16.971535 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:16.971839 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:16.974997 (+ 3158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:16.975026 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:16.975128 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:01:16.976075 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.976126 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.977983 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.978005 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.980624 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:16.980682 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:16.983113 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:16.983149 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:16.983651 (+ 502us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:16.983804 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 03:01:16.983959 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 03:01:16.984256 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17709,"mutex_wait_us":167,"thread_start_us":10814,"threads_started":1}
I20260812 03:01:16.990032 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44660217c0354a2a996e731e1aeb7c6d" candidate_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4" is_pre_election: true
I20260812 03:01:16.990594 22940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44660217c0354a2a996e731e1aeb7c6d" candidate_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc87127101a24af78b78da0570426657" is_pre_election: true
I20260812 03:01:16.990793 22688 raft_consensus.cc:2454] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d09f723a9a2408c8a3c98a88bb8ca56 in term 0.
I20260812 03:01:16.991353 22940 raft_consensus.cc:2454] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d09f723a9a2408c8a3c98a88bb8ca56 in term 0.
I20260812 03:01:16.992033 22315 leader_election.cc:302] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09f723a9a2408c8a3c98a88bb8ca56, f04caa54d700475697a9dcfe72b8d7b4; no voters:
I20260812 03:01:16.992961 23251 raft_consensus.cc:2781] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:01:16.993234 23251 raft_consensus.cc:482] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:01:16.993508 23251 raft_consensus.cc:3037] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:16.998878 23251 raft_consensus.cc:504] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.000675 23251 leader_election.cc:288] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [CANDIDATE]: Term 1 election: Requested vote from peers f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553), fc87127101a24af78b78da0570426657 (127.10.118.67:42869)
I20260812 03:01:17.001639 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44660217c0354a2a996e731e1aeb7c6d" candidate_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4"
I20260812 03:01:17.001797 22940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44660217c0354a2a996e731e1aeb7c6d" candidate_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc87127101a24af78b78da0570426657"
I20260812 03:01:17.002110 22688 raft_consensus.cc:3037] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:17.002345 22940 raft_consensus.cc:3037] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:17.009302 22688 raft_consensus.cc:2454] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d09f723a9a2408c8a3c98a88bb8ca56 in term 1.
I20260812 03:01:17.010178 22315 leader_election.cc:302] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09f723a9a2408c8a3c98a88bb8ca56, f04caa54d700475697a9dcfe72b8d7b4; no voters:
I20260812 03:01:17.010145 22940 raft_consensus.cc:2454] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d09f723a9a2408c8a3c98a88bb8ca56 in term 1.
I20260812 03:01:17.011080 23251 raft_consensus.cc:2781] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:17.012161 23251 raft_consensus.cc:686] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 LEADER]: Becoming Leader. State: Replica: 8d09f723a9a2408c8a3c98a88bb8ca56, State: Running, Role: LEADER
I20260812 03:01:17.013500 23251 consensus_queue.cc:237] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.020908 22147 catalog_manager.cc:5495] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65). New cstate: current_term: 1 leader_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } health_report { overall_health: UNKNOWN } } }
I20260812 03:01:17.077014 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:17.066385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.10.118.65:42009)
0812 03:01:17.068293 (+ 1908us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:17.068314 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:17.068349 (+ 35us) server_negotiation.cc:407] Connection header received
0812 03:01:17.068557 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:17.068576 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:17.068811 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:17.069001 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:01:17.069892 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:17.071022 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:17.072877 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:17.073503 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:17.073790 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:17.074577 (+ 787us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:01:17.075077 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:01:17.075266 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:17.075545 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 03:01:17.075746 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1574,"thread_start_us":1298,"threads_started":1}
I20260812 03:01:17.085700 22688 raft_consensus.cc:1261] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Refusing update from remote peer 8d09f723a9a2408c8a3c98a88bb8ca56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:17.085837 22940 raft_consensus.cc:1261] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Refusing update from remote peer 8d09f723a9a2408c8a3c98a88bb8ca56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:17.087011 23251 consensus_queue.cc:1035] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:01:17.087900 23259 consensus_queue.cc:1035] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 03:01:17.140560 22148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:17.141322 22148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:17.151114 22148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Sending DeleteTablet for 3 replicas of tablet 44660217c0354a2a996e731e1aeb7c6d
I20260812 03:01:17.153283 22652 tablet_service.cc:1514] Processing DeleteTablet for tablet 44660217c0354a2a996e731e1aeb7c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:17 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:42494
I20260812 03:01:17.153319 22407 tablet_service.cc:1514] Processing DeleteTablet for tablet 44660217c0354a2a996e731e1aeb7c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:17 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57308
I20260812 03:01:17.160583 22905 tablet_service.cc:1514] Processing DeleteTablet for tablet 44660217c0354a2a996e731e1aeb7c6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:17 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:59546
I20260812 03:01:17.164600 23283 tablet_replica.cc:330] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: stopping tablet replica
I20260812 03:01:17.166527 23283 raft_consensus.cc:2229] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:01:17.167539 22147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 03:01:17.169580 23281 tablet_replica.cc:330] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: stopping tablet replica
W20260812 03:01:17.169705 22147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:01:17.170459 23283 raft_consensus.cc:2258] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:17.171069 23280 tablet_replica.cc:330] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: stopping tablet replica
I20260812 03:01:17.173399 23280 raft_consensus.cc:2229] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:01:17.176142 23283 ts_tablet_manager.cc:1905] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:17.172194 23281 raft_consensus.cc:2229] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:01:17.177423 23281 raft_consensus.cc:2258] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:17.178614 23280 raft_consensus.cc:2258] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:17.193107 22407 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6272efa7cc4ed98188eeeacbfae5a3 (DEFAULT_TABLE table=test-table [id=432206add36b4b5195e516810318f730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:17.193491 23280 ts_tablet_manager.cc:1905] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:17.194257 22407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6272efa7cc4ed98188eeeacbfae5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:17.195040 22652 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6272efa7cc4ed98188eeeacbfae5a3 (DEFAULT_TABLE table=test-table [id=432206add36b4b5195e516810318f730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:17.196712 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6272efa7cc4ed98188eeeacbfae5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:17.202387 22905 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6272efa7cc4ed98188eeeacbfae5a3 (DEFAULT_TABLE table=test-table [id=432206add36b4b5195e516810318f730]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:01:17.203495 22905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a6272efa7cc4ed98188eeeacbfae5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:17.218246 23283 ts_tablet_manager.cc:1918] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:01:17.222506 23283 log.cc:1196] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/44660217c0354a2a996e731e1aeb7c6d
I20260812 03:01:17.223270 23283 ts_tablet_manager.cc:1939] T 44660217c0354a2a996e731e1aeb7c6d P fc87127101a24af78b78da0570426657: Deleting consensus metadata
I20260812 03:01:17.230957 23280 ts_tablet_manager.cc:1918] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:01:17.233647 22124 catalog_manager.cc:4841] TS fc87127101a24af78b78da0570426657 (127.10.118.67:42869): tablet 44660217c0354a2a996e731e1aeb7c6d (table test-table [id=e165f979846e4753b0e51bd5225368ce]) successfully deleted
I20260812 03:01:17.228240 23281 ts_tablet_manager.cc:1905] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:17.236457 23240 tablet_bootstrap.cc:492] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56: Bootstrap starting.
I20260812 03:01:17.239852 23280 log.cc:1196] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-0/wal/wals/44660217c0354a2a996e731e1aeb7c6d
I20260812 03:01:17.240687 23280 ts_tablet_manager.cc:1939] T 44660217c0354a2a996e731e1aeb7c6d P 8d09f723a9a2408c8a3c98a88bb8ca56: Deleting consensus metadata
I20260812 03:01:17.243271 22125 catalog_manager.cc:4841] TS 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009): tablet 44660217c0354a2a996e731e1aeb7c6d (table test-table [id=e165f979846e4753b0e51bd5225368ce]) successfully deleted
I20260812 03:01:17.246124 23240 tablet_bootstrap.cc:654] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:17.248116 23237 tablet_bootstrap.cc:492] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: Bootstrap starting.
I20260812 03:01:17.254076 23236 tablet_bootstrap.cc:492] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: Bootstrap starting.
I20260812 03:01:17.255409 23237 tablet_bootstrap.cc:654] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:17.259974 23236 tablet_bootstrap.cc:654] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:17.269150 23236 tablet_bootstrap.cc:492] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: No bootstrap required, opened a new log
I20260812 03:01:17.269662 23236 ts_tablet_manager.cc:1397] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.001s
I20260812 03:01:17.271633 23237 tablet_bootstrap.cc:492] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: No bootstrap required, opened a new log
I20260812 03:01:17.272073 23237 ts_tablet_manager.cc:1397] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.001s
I20260812 03:01:17.272465 23236 raft_consensus.cc:348] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.273283 23236 raft_consensus.cc:374] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:17.273620 23236 raft_consensus.cc:729] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f04caa54d700475697a9dcfe72b8d7b4, State: Initialized, Role: FOLLOWER
I20260812 03:01:17.274386 23236 consensus_queue.cc:260] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.274521 23237 raft_consensus.cc:348] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.275187 23237 raft_consensus.cc:374] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:17.275449 23237 raft_consensus.cc:729] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc87127101a24af78b78da0570426657, State: Initialized, Role: FOLLOWER
I20260812 03:01:17.276156 23237 consensus_queue.cc:260] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.285331 23237 ts_tablet_manager.cc:1428] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260812 03:01:17.289419 23281 ts_tablet_manager.cc:1918] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:01:17.289875 23281 log.cc:1196] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/44660217c0354a2a996e731e1aeb7c6d
I20260812 03:01:17.290642 23281 ts_tablet_manager.cc:1939] T 44660217c0354a2a996e731e1aeb7c6d P f04caa54d700475697a9dcfe72b8d7b4: Deleting consensus metadata
I20260812 03:01:17.293362 23236 ts_tablet_manager.cc:1428] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20260812 03:01:17.293630 23240 tablet_bootstrap.cc:492] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56: No bootstrap required, opened a new log
I20260812 03:01:17.294073 23240 ts_tablet_manager.cc:1397] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.014s
I20260812 03:01:17.296769 23240 raft_consensus.cc:348] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.297609 23240 raft_consensus.cc:374] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:17.297941 23240 raft_consensus.cc:729] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d09f723a9a2408c8a3c98a88bb8ca56, State: Initialized, Role: FOLLOWER
I20260812 03:01:17.298728 23240 consensus_queue.cc:260] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.301445 22122 catalog_manager.cc:4841] TS f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553): tablet 44660217c0354a2a996e731e1aeb7c6d (table test-table [id=e165f979846e4753b0e51bd5225368ce]) successfully deleted
I20260812 03:01:17.303344 23240 ts_tablet_manager.cc:1428] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260812 03:01:17.391278 23193 raft_consensus.cc:482] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:01:17.391793 23193 raft_consensus.cc:504] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.393898 23193 leader_election.cc:288] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009), f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553)
I20260812 03:01:17.395860 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6272efa7cc4ed98188eeeacbfae5a3" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4" is_pre_election: true
I20260812 03:01:17.396565 22688 raft_consensus.cc:2454] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:17.397982 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6272efa7cc4ed98188eeeacbfae5a3" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" is_pre_election: true
I20260812 03:01:17.398609 22446 raft_consensus.cc:2454] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 0.
I20260812 03:01:17.400049 22809 leader_election.cc:302] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f04caa54d700475697a9dcfe72b8d7b4, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:17.401845 23193 raft_consensus.cc:2781] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:01:17.402163 23193 raft_consensus.cc:482] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:01:17.403983 23193 raft_consensus.cc:3037] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:17.410866 23193 raft_consensus.cc:504] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.413404 22446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6272efa7cc4ed98188eeeacbfae5a3" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56"
I20260812 03:01:17.413982 22446 raft_consensus.cc:3037] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:17.422837 23193 leader_election.cc:288] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Requested vote from peers 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009), f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553)
I20260812 03:01:17.424190 22688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a6272efa7cc4ed98188eeeacbfae5a3" candidate_uuid: "fc87127101a24af78b78da0570426657" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f04caa54d700475697a9dcfe72b8d7b4"
I20260812 03:01:17.424817 22688 raft_consensus.cc:3037] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:17.426481 22446 raft_consensus.cc:2454] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:17.427706 22813 leader_election.cc:302] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d09f723a9a2408c8a3c98a88bb8ca56, fc87127101a24af78b78da0570426657; no voters:
I20260812 03:01:17.428699 23193 raft_consensus.cc:2781] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:17.431707 22688 raft_consensus.cc:2454] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc87127101a24af78b78da0570426657 in term 1.
I20260812 03:01:17.433550 23193 raft_consensus.cc:686] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 1 LEADER]: Becoming Leader. State: Replica: fc87127101a24af78b78da0570426657, State: Running, Role: LEADER
I20260812 03:01:17.434415 23193 consensus_queue.cc:237] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } }
I20260812 03:01:17.442361 22147 catalog_manager.cc:5495] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc87127101a24af78b78da0570426657 (127.10.118.67). New cstate: current_term: 1 leader_uuid: "fc87127101a24af78b78da0570426657" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc87127101a24af78b78da0570426657" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42869 } health_report { overall_health: HEALTHY } } }
I20260812 03:01:17.536734 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:17.519576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.10.118.67:42869)
0812 03:01:17.525815 (+ 6239us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:17.525837 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:17.525883 (+ 46us) server_negotiation.cc:407] Connection header received
0812 03:01:17.526033 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:17.526059 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:17.526360 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:17.526529 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:01:17.527493 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:17.529110 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:17.531846 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:17.532805 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:17.533082 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:17.534069 (+ 987us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:01:17.534340 (+ 271us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:01:17.534480 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:17.534843 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 03:01:17.535023 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5792,"thread_start_us":5511,"threads_started":1}
I20260812 03:01:17.566383 22446 raft_consensus.cc:1261] T 4a6272efa7cc4ed98188eeeacbfae5a3 P 8d09f723a9a2408c8a3c98a88bb8ca56 [term 1 FOLLOWER]: Refusing update from remote peer fc87127101a24af78b78da0570426657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:17.566546 22688 raft_consensus.cc:1261] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Refusing update from remote peer fc87127101a24af78b78da0570426657: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:01:17.567767 23292 consensus_queue.cc:1035] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d09f723a9a2408c8a3c98a88bb8ca56" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 42009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:01:17.568444 23193 consensus_queue.cc:1035] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f04caa54d700475697a9dcfe72b8d7b4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 03:01:17.582181 23012 debug-util.cc:398] Leaking SignalData structure 0x7b08000e7080 after lost signal to thread 22772
I20260812 03:01:17.585202 23273 mvcc.cc:205] Tried to move back new op lower bound from 7317519063290540032 to 7317519062779772928. Current Snapshot: MvccSnapshot[applied={T|T < 7317519063290540032}]
I20260812 03:01:17.850163 22147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:17.850785 22147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:01:17.862150 22147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebb6d41de8f4ee2af6d9cf7fc163f21: Sending DeleteTablet for 3 replicas of tablet 4a6272efa7cc4ed98188eeeacbfae5a3
I20260812 03:01:17.863910 22407 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6272efa7cc4ed98188eeeacbfae5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:17 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57308
I20260812 03:01:17.868319 22652 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6272efa7cc4ed98188eeeacbfae5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:17 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:42494
I20260812 03:01:17.870488 22905 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a6272efa7cc4ed98188eeeacbfae5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:01:17 UTC) from {username='oryx', principal='oryx/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:59546
I20260812 03:01:17.873135 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 22283
I20260812 03:01:17.886513 23315 tablet_replica.cc:330] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: stopping tablet replica
I20260812 03:01:17.887639 23315 raft_consensus.cc:2229] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:01:17.894234 23311 tablet_replica.cc:330] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: stopping tablet replica
I20260812 03:01:17.894498 23315 raft_consensus.cc:2258] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:17.894995 23311 raft_consensus.cc:2229] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:01:17.895495 23311 raft_consensus.cc:2258] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:01:17.898181 23311 ts_tablet_manager.cc:1905] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:17.912722 23311 ts_tablet_manager.cc:1918] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 03:01:17.913074 23311 log.cc:1196] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/4a6272efa7cc4ed98188eeeacbfae5a3
I20260812 03:01:17.913789 23311 ts_tablet_manager.cc:1939] T 4a6272efa7cc4ed98188eeeacbfae5a3 P f04caa54d700475697a9dcfe72b8d7b4: Deleting consensus metadata
I20260812 03:01:17.916043 23315 ts_tablet_manager.cc:1905] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:01:17.916337 22122 catalog_manager.cc:4841] TS f04caa54d700475697a9dcfe72b8d7b4 (127.10.118.66:45553): tablet 4a6272efa7cc4ed98188eeeacbfae5a3 (table test-table [id=432206add36b4b5195e516810318f730]) successfully deleted
I20260812 03:01:17.930356 23315 ts_tablet_manager.cc:1918] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 03:01:17.930757 23315 log.cc:1196] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/4a6272efa7cc4ed98188eeeacbfae5a3
I20260812 03:01:17.931560 23315 ts_tablet_manager.cc:1939] T 4a6272efa7cc4ed98188eeeacbfae5a3 P fc87127101a24af78b78da0570426657: Deleting consensus metadata
I20260812 03:01:17.934147 22124 catalog_manager.cc:4841] TS fc87127101a24af78b78da0570426657 (127.10.118.67:42869): tablet 4a6272efa7cc4ed98188eeeacbfae5a3 (table test-table [id=432206add36b4b5195e516810318f730]) successfully deleted
W20260812 03:01:17.959731 22125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:42009 (error 108)
I20260812 03:01:17.973315 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 22526
I20260812 03:01:18.007187 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:17.973522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:42009 (local address 127.0.0.1:50798)
0812 03:01:18.005947 (+ 32425us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:18.006287 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:42009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31757,"threads_started":1}
W20260812 03:01:18.008162 22125 catalog_manager.cc:4584] TS 8d09f723a9a2408c8a3c98a88bb8ca56 (127.10.118.65:42009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a6272efa7cc4ed98188eeeacbfae5a3: Network error: Client connection negotiation failed: client connection to 127.10.118.65:42009: connect: Connection refused (error 111)
I20260812 03:01:18.040418 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 22769
I20260812 03:01:18.063410 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:18.062371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:42009 (local address 127.0.0.1:50804)
0812 03:01:18.062838 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:18.062996 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:42009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 03:01:18.065793 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:18.064821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:42009 (local address 127.0.0.1:50808)
0812 03:01:18.065255 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:18.065409 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:42009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 03:01:18.092953 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 22089
2026-08-12T03:01:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (13179 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:01:18 dist-test-slave-zpvx krb5kdc[23321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:01:18 dist-test-slave-zpvx krb5kdc[23321](info): set up 2 sockets
Aug 12 03:01:18 dist-test-slave-zpvx krb5kdc[23321](info): commencing operation
krb5kdc: starting...
W20260812 03:01:19.539645 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.367s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:01:19 dist-test-slave-zpvx krb5kdc[23321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:01:19.781103 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43293
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:43293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:20.088160 23344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:20.088833 23344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:20.089475 23344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:20.120525 23344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:01:20.120894 23344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:20.121236 23344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:20.121502 23344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:01:20.121757 23344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:01:20.158013 23344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:43293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:20.159238 23344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:20.160303 23344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:20.177949 23356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:20.178087 23360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:20.178094 23358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:20.179342 23344 server_base.cc:1034] running on GCE node
I20260812 03:01:21.370389 23344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:21.373555 23344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:21.374969 23344 hybrid_clock.cc:648] HybridClock initialized: now 1786503681374946 us; error 46 us; skew 500 ppm
I20260812 03:01:21.375846 23344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T03:01:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (4279 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:01:22 dist-test-slave-zpvx krb5kdc[23374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:01:22 dist-test-slave-zpvx krb5kdc[23374](info): set up 2 sockets
Aug 12 03:01:22 dist-test-slave-zpvx krb5kdc[23374](info): commencing operation
krb5kdc: starting...
W20260812 03:01:23.789520 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:01:23 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:01:24.017405 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41479
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:24.325466 23402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:24.326100 23402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:24.326721 23402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:24.357360 23402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:01:24.357717 23402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:24.358024 23402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:24.358287 23402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:01:24.358533 23402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:01:24.394438 23402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:24.395660 23402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:24.396775 23402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:24.413882 23415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:24.413942 23413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:24.414942 23417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:25.588034 23416 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:01:25.588165 23402 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:01:25.592190 23402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:25.594877 23402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:25.596243 23402 hybrid_clock.cc:648] HybridClock initialized: now 1786503685596217 us; error 47 us; skew 500 ppm
I20260812 03:01:25.597122 23402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:25 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503685, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:25.603356 23402 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:01:25.608793 23402 webserver.cc:460] Webserver started at http://127.10.118.126:36145/ using document root <none> and password file <none>
I20260812 03:01:25.609740 23402 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:25.609987 23402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:25.610496 23402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:25.615629 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "fe4a1fe1bce8473cbc4347baecdaabe2"
format_stamp: "Formatted at 2026-08-12 03:01:25 on dist-test-slave-zpvx"
server_key: "a66b9d77d5829ed83faf7896cb6ad1d6"
server_key_iv: "f750fd2b5717b3cdc313026a83741892"
server_key_version: "encryptionkey@0"
I20260812 03:01:25.616895 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "fe4a1fe1bce8473cbc4347baecdaabe2"
format_stamp: "Formatted at 2026-08-12 03:01:25 on dist-test-slave-zpvx"
server_key: "a66b9d77d5829ed83faf7896cb6ad1d6"
server_key_iv: "f750fd2b5717b3cdc313026a83741892"
server_key_version: "encryptionkey@0"
I20260812 03:01:25.624156 23402 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:01:25.629698 23432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:25.630699 23402 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 03:01:25.631027 23402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "fe4a1fe1bce8473cbc4347baecdaabe2"
format_stamp: "Formatted at 2026-08-12 03:01:25 on dist-test-slave-zpvx"
server_key: "a66b9d77d5829ed83faf7896cb6ad1d6"
server_key_iv: "f750fd2b5717b3cdc313026a83741892"
server_key_version: "encryptionkey@0"
I20260812 03:01:25.631398 23402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:25.687876 23402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:25.693553 23402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:25.694051 23402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:25.763695 23402 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:41479
I20260812 03:01:25.763777 23518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:41479 every 8 connection(s)
I20260812 03:01:25.766470 23402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:01:25.771704 23519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:25.772558 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 23402
I20260812 03:01:25.773001 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:01:25.774230 10713 external_mini_cluster.cc:1428] Setting key 8c41b75dffa8b4f2158552bce140fbfc
I20260812 03:01:25.798266 23519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Bootstrap starting.
Aug 12 03:01:25 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:01:25.805234 23519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:25.807693 23519 log.cc:824] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Log is configured to *not* fsync() on all Append() calls
I20260812 03:01:25.813696 23519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: No bootstrap required, opened a new log
I20260812 03:01:25.813865 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:25.776824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.10.118.126:41479)
0812 03:01:25.779364 (+ 2540us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:25.779417 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:25.779490 (+ 73us) server_negotiation.cc:407] Connection header received
0812 03:01:25.784974 (+ 5484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:25.785035 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:25.785544 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:25.786034 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:25.788650 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:25.791203 (+ 2553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:25.793358 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:25.794723 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:25.795084 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:25.800535 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:25.800590 (+ 55us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:25.800636 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:25.800800 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:25.806776 (+ 5976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:25.808120 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:25.808147 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:25.808163 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:25.808487 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:25.809189 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:25.809210 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:25.809224 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:25.809931 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:25.810486 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:25.811212 (+ 726us) server_negotiation.cc:299] Negotiation successful
0812 03:01:25.811864 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"thread_start_us":1612,"threads_started":1}
I20260812 03:01:25.836031 23519 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:25.836795 23519 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:25.837064 23519 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4a1fe1bce8473cbc4347baecdaabe2, State: Initialized, Role: FOLLOWER
I20260812 03:01:25.837795 23519 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:25.838297 23519 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:01:25.838579 23519 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:01:25.838900 23519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:25.843816 23519 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:25.844530 23519 leader_election.cc:302] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4a1fe1bce8473cbc4347baecdaabe2; no voters:
I20260812 03:01:25.846200 23519 leader_election.cc:288] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:01:25.846601 23530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:25.848832 23530 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 LEADER]: Becoming Leader. State: Replica: fe4a1fe1bce8473cbc4347baecdaabe2, State: Running, Role: LEADER
I20260812 03:01:25.849691 23530 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:25.850571 23519 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:01:25.860839 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } } }
I20260812 03:01:25.861783 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:25.861780 23533 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4a1fe1bce8473cbc4347baecdaabe2. Latest consensus state: current_term: 1 leader_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } } }
I20260812 03:01:25.862690 23533 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:25.866024 23548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:01:25.879035 23548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:01:25.894622 23548 catalog_manager.cc:1347] Generated new cluster ID: 61630520c2f949b4854847c453de2602
I20260812 03:01:25.894932 23548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:01:25.915450 23548 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:01:25.917375 23548 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:01:25.928009 23548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Generated new TSK 0
I20260812 03:01:25.929113 23548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:01:26.014325 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41479
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:26.343815 23575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:26.344436 23575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:26.345112 23575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:26.377556 23575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:26.377921 23575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:26.378912 23575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:01:26.414566 23575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:26.415756 23575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:26.416879 23575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:26.437273 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:26.439003 23594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:26.437285 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:27.697085 23592 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1257 milliseconds
I20260812 03:01:27.697207 23575 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:01:27.698475 23575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:27.700891 23575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:27.702292 23575 hybrid_clock.cc:648] HybridClock initialized: now 1786503687702259 us; error 51 us; skew 500 ppm
I20260812 03:01:27.703104 23575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:27 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:27.715126 23575 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:01:27.721149 23575 webserver.cc:460] Webserver started at http://127.10.118.65:46081/ using document root <none> and password file <none>
I20260812 03:01:27.722070 23575 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:27.722295 23575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:27.722767 23575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:27.727990 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "48f6355857bd4fcda480ed0fdb69ba4b"
format_stamp: "Formatted at 2026-08-12 03:01:27 on dist-test-slave-zpvx"
server_key: "f6e0ff903612311c796189adcf41dbed"
server_key_iv: "008fd4c32e89febe9aa94130c5226d2c"
server_key_version: "encryptionkey@0"
I20260812 03:01:27.729301 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "48f6355857bd4fcda480ed0fdb69ba4b"
format_stamp: "Formatted at 2026-08-12 03:01:27 on dist-test-slave-zpvx"
server_key: "f6e0ff903612311c796189adcf41dbed"
server_key_iv: "008fd4c32e89febe9aa94130c5226d2c"
server_key_version: "encryptionkey@0"
I20260812 03:01:27.737126 23575 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:01:27.743490 23612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:27.744738 23575 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.001s
I20260812 03:01:27.745067 23575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "48f6355857bd4fcda480ed0fdb69ba4b"
format_stamp: "Formatted at 2026-08-12 03:01:27 on dist-test-slave-zpvx"
server_key: "f6e0ff903612311c796189adcf41dbed"
server_key_iv: "008fd4c32e89febe9aa94130c5226d2c"
server_key_version: "encryptionkey@0"
I20260812 03:01:27.745445 23575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:27.790513 23575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:27.796258 23575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:27.796769 23575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:27.799882 23575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:27.804119 23575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:27.804322 23575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:27.804661 23575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:27.804812 23575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:27.975976 23575 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:32987
I20260812 03:01:27.976163 23809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:32987 every 8 connection(s)
I20260812 03:01:27.978607 23575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:01:27.984453 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 23575
I20260812 03:01:27.985060 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:01:27.986974 10713 external_mini_cluster.cc:1428] Setting key dccad5ba1c381b36534ba387e56bf1c7
Aug 12 03:01:28 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503687, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:01:28.032950 23817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:27.984003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:34177 (local address 127.10.118.126:41479)
0812 03:01:27.999672 (+ 15669us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:27.999698 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:27.999757 (+ 59us) server_negotiation.cc:407] Connection header received
0812 03:01:28.004296 (+ 4539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:28.004333 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:28.004562 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:28.004750 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:28.007108 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:28.009078 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:28.012293 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:28.013426 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:28.013806 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:28.020528 (+ 6722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:28.020550 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:28.020570 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:28.020666 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:28.025517 (+ 4851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:28.027101 (+ 1584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:28.027136 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:28.027164 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:28.027415 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:28.028318 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:28.028353 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:28.028408 (+ 55us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:28.029195 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:28.029462 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:28.030809 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0812 03:01:28.031322 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15208,"thread_start_us":15682,"threads_started":1}
I20260812 03:01:28.034529 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:27.984276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:34177)
0812 03:01:27.996664 (+ 12388us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:27.996786 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:28.003917 (+ 7131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:28.005041 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:28.005083 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:28.005951 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:28.006822 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:28.006866 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:28.009377 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:28.009411 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:28.012001 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:28.012054 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:28.013809 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:28.013842 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:28.014362 (+ 520us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:28.015506 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:28.015603 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:28.020065 (+ 4462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:28.025791 (+ 5726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:28.025825 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:28.025857 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:28.026864 (+ 1007us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:28.027658 (+ 794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:28.027686 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:28.027719 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:28.028082 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:28.029458 (+ 1376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:28.029501 (+ 43us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:28.030101 (+ 600us) client_negotiation.cc:769] Sending connection context
0812 03:01:28.030552 (+ 451us) client_negotiation.cc:240] Negotiation successful
0812 03:01:28.031261 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11660,"thread_start_us":15394,"threads_started":1}
I20260812 03:01:28.036883 23814 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41479
I20260812 03:01:28.037384 23814 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:28.038643 23814 heartbeater.cc:511] Master 127.10.118.126:41479 requested a full tablet report, sending...
I20260812 03:01:28.042209 23461 ts_manager.cc:194] Registered new tserver with Master: 48f6355857bd4fcda480ed0fdb69ba4b (127.10.118.65:32987)
I20260812 03:01:28.045187 23461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:34177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:01:28.056538 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41479
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:28.386086 23822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:28.386685 23822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:28.387421 23822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:28.423804 23822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:28.424196 23822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:28.425251 23822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:01:28.461807 23822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:28.463042 23822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:28.464105 23822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:28.484126 23831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:29.049525 23814 heartbeater.cc:503] Master 127.10.118.126:41479 was elected leader, sending a full tablet report...
W20260812 03:01:28.484263 23836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:28.486613 23822 server_base.cc:1034] running on GCE node
W20260812 03:01:28.485283 23832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:29.650938 23822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:29.653385 23822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:29.654724 23822 hybrid_clock.cc:648] HybridClock initialized: now 1786503689654684 us; error 51 us; skew 500 ppm
I20260812 03:01:29.655540 23822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:29 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503689, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:29.661602 23822 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:01:29.666931 23822 webserver.cc:460] Webserver started at http://127.10.118.66:37479/ using document root <none> and password file <none>
I20260812 03:01:29.668000 23822 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:29.668234 23822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:29.668716 23822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:29.673941 23822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "f28b7849e7524cb39de614ef207d4609"
format_stamp: "Formatted at 2026-08-12 03:01:29 on dist-test-slave-zpvx"
server_key: "42832ac630c7754abd68da9c414b0d01"
server_key_iv: "07e2553c940433aef268c33e153e7b09"
server_key_version: "encryptionkey@0"
I20260812 03:01:29.675099 23822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "f28b7849e7524cb39de614ef207d4609"
format_stamp: "Formatted at 2026-08-12 03:01:29 on dist-test-slave-zpvx"
server_key: "42832ac630c7754abd68da9c414b0d01"
server_key_iv: "07e2553c940433aef268c33e153e7b09"
server_key_version: "encryptionkey@0"
I20260812 03:01:29.682301 23822 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260812 03:01:29.687813 23847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:29.689043 23822 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:01:29.689388 23822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "f28b7849e7524cb39de614ef207d4609"
format_stamp: "Formatted at 2026-08-12 03:01:29 on dist-test-slave-zpvx"
server_key: "42832ac630c7754abd68da9c414b0d01"
server_key_iv: "07e2553c940433aef268c33e153e7b09"
server_key_version: "encryptionkey@0"
I20260812 03:01:29.689760 23822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:29.733764 23822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:29.739764 23822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:29.740268 23822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:29.742849 23822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:29.747287 23822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:29.747481 23822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:29.747781 23822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:29.747928 23822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:29.889907 23822 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:35977
I20260812 03:01:29.890010 24062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:35977 every 8 connection(s)
I20260812 03:01:29.892505 23822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:01:29.897964 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 23822
I20260812 03:01:29.898599 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:01:29.900079 10713 external_mini_cluster.cc:1428] Setting key 68a900ec1aed5f609742f0b66b61272b
Aug 12 03:01:29 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503689, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:01:29.928020 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:29.896683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:50135 (local address 127.10.118.126:41479)
0812 03:01:29.898399 (+ 1716us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:29.898421 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:29.900100 (+ 1679us) server_negotiation.cc:407] Connection header received
0812 03:01:29.905319 (+ 5219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:29.905354 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:29.905538 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:29.905717 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:29.908028 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:29.909139 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:29.911346 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:29.911928 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:29.912219 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:29.917131 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:29.917169 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:29.917202 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:29.917360 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:29.921073 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:29.922241 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:29.922271 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:29.922284 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:29.922482 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:29.923198 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:29.923218 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:29.923232 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:29.923722 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:29.923997 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:29.925230 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0812 03:01:29.925710 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337,"thread_start_us":999,"threads_started":1}
I20260812 03:01:29.928887 24068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:29.896853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:50135)
0812 03:01:29.899850 (+ 2997us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:29.899933 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:29.904968 (+ 5035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:29.905999 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:29.906038 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:29.906887 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:29.907752 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:29.907795 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:29.909387 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:29.909407 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:29.911090 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:29.911130 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:29.912268 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:29.912315 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:29.912773 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:29.913681 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:29.913749 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:29.916776 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:29.921310 (+ 4534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:29.921345 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:29.921368 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:29.922037 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:29.922671 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:29.922701 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:29.922718 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:29.922990 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:29.923969 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:29.924011 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:29.924529 (+ 518us) client_negotiation.cc:769] Sending connection context
0812 03:01:29.925021 (+ 492us) client_negotiation.cc:240] Negotiation successful
0812 03:01:29.925719 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2575,"thread_start_us":1830,"threads_started":1}
I20260812 03:01:29.930852 24064 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41479
I20260812 03:01:29.931346 24064 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:29.932628 24064 heartbeater.cc:511] Master 127.10.118.126:41479 requested a full tablet report, sending...
I20260812 03:01:29.935053 23461 ts_manager.cc:194] Registered new tserver with Master: f28b7849e7524cb39de614ef207d4609 (127.10.118.66:35977)
I20260812 03:01:29.936537 23461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:50135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:01:29.956990 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41479
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:30.269124 24074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:30.269680 24074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:30.270328 24074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:30.301316 24074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:30.301712 24074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:30.302646 24074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:01:30.339967 24074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:30.341199 24074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:30.342268 24074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:30.360806 24087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:30.939942 24064 heartbeater.cc:503] Master 127.10.118.126:41479 was elected leader, sending a full tablet report...
W20260812 03:01:30.361708 24088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:30.361129 24090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:30.362622 24074 server_base.cc:1034] running on GCE node
I20260812 03:01:31.555075 24074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:31.557610 24074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:31.558960 24074 hybrid_clock.cc:648] HybridClock initialized: now 1786503691558921 us; error 58 us; skew 500 ppm
I20260812 03:01:31.559789 24074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:31 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503691, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:31.566157 24074 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:01:31.571646 24074 webserver.cc:460] Webserver started at http://127.10.118.67:45535/ using document root <none> and password file <none>
I20260812 03:01:31.572633 24074 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:31.572876 24074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:31.573351 24074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:31.578580 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "24d281942e1b48a1a84a03bd6e55b3ee"
format_stamp: "Formatted at 2026-08-12 03:01:31 on dist-test-slave-zpvx"
server_key: "07c53cc019a7ca3f66299eb8c8289a76"
server_key_iv: "4e0d66c27f82f60382842701e60f025a"
server_key_version: "encryptionkey@0"
I20260812 03:01:31.579749 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "24d281942e1b48a1a84a03bd6e55b3ee"
format_stamp: "Formatted at 2026-08-12 03:01:31 on dist-test-slave-zpvx"
server_key: "07c53cc019a7ca3f66299eb8c8289a76"
server_key_iv: "4e0d66c27f82f60382842701e60f025a"
server_key_version: "encryptionkey@0"
I20260812 03:01:31.586982 24074 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 03:01:31.592620 24102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:31.593686 24074 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:01:31.594012 24074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "24d281942e1b48a1a84a03bd6e55b3ee"
format_stamp: "Formatted at 2026-08-12 03:01:31 on dist-test-slave-zpvx"
server_key: "07c53cc019a7ca3f66299eb8c8289a76"
server_key_iv: "4e0d66c27f82f60382842701e60f025a"
server_key_version: "encryptionkey@0"
I20260812 03:01:31.594403 24074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:31.638945 24074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:31.644835 24074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:31.645310 24074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:31.647923 24074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:31.652573 24074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:31.652771 24074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:31.653074 24074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:31.653241 24074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:31.790648 24074 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:44801
I20260812 03:01:31.790827 24323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:44801 every 8 connection(s)
I20260812 03:01:31.793287 24074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:01:31.795722 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 24074
I20260812 03:01:31.796356 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:01:31.798084 10713 external_mini_cluster.cc:1428] Setting key 2def16ea338de0154c03b492e202b05c
Aug 12 03:01:31 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503691, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:01:31.828442 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:31.797915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:46847 (local address 127.10.118.126:41479)
0812 03:01:31.799195 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:31.799221 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:31.801417 (+ 2196us) server_negotiation.cc:407] Connection header received
0812 03:01:31.806770 (+ 5353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:31.806791 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:31.806963 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:31.807109 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:31.809472 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:31.810560 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:31.812625 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:31.813307 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:31.813495 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:31.819184 (+ 5689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:31.819209 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:31.819225 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:31.819322 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:31.823083 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:31.824142 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:31.824162 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:31.824175 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:31.824364 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:31.825033 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:31.825051 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:31.825064 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:31.825540 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:31.825787 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:31.826666 (+ 879us) server_negotiation.cc:299] Negotiation successful
0812 03:01:31.826981 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":540,"threads_started":1}
I20260812 03:01:31.829119 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:31.798159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:46847)
0812 03:01:31.801140 (+ 2981us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:31.801258 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:31.806384 (+ 5126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:31.807357 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:31.807380 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:31.808158 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:31.809210 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:31.809257 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:31.810791 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:31.810812 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:31.812348 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:31.812450 (+ 102us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:31.813582 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:31.813609 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:31.813973 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:31.815127 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:31.815222 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:31.818875 (+ 3653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:31.823270 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:31.823293 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:31.823322 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:31.823955 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:31.824583 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:01:31.824602 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:01:31.824616 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:01:31.824860 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:01:31.825734 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:31.825762 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:31.826153 (+ 391us) client_negotiation.cc:769] Sending connection context
0812 03:01:31.826465 (+ 312us) client_negotiation.cc:240] Negotiation successful
0812 03:01:31.826905 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2406,"thread_start_us":1545,"threads_started":1}
I20260812 03:01:31.830961 24324 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41479
I20260812 03:01:31.831420 24324 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:31.832571 24324 heartbeater.cc:511] Master 127.10.118.126:41479 requested a full tablet report, sending...
I20260812 03:01:31.834535 23461 ts_manager.cc:194] Registered new tserver with Master: 24d281942e1b48a1a84a03bd6e55b3ee (127.10.118.67:44801)
I20260812 03:01:31.835929 23461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:46847
I20260812 03:01:31.845814 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab.
Entry for principal oryx/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab.
WARNING: no policy specified for oryx/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.118.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab.
Entry for principal oryx/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab.
WARNING: no policy specified for oryx/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.118.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab.
Entry for principal oryx/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab.
I20260812 03:01:31.999742 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 23575
I20260812 03:01:32.040472 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 23822
I20260812 03:01:32.072326 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 24074
I20260812 03:01:32.116498 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 23402
I20260812 03:01:32.153256 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41479
--webserver_interface=127.10.118.126
--webserver_port=36145
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:41479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:01:32.463531 24343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:32.464159 24343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:32.464810 24343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:32.495163 24343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:01:32.495548 24343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:32.495859 24343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:32.496122 24343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:01:32.496397 24343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:01:32.532461 24343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:41479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=36145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:32.533706 24343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:32.534761 24343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:32.552317 24359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:32.552417 24356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:32.553347 24343 server_base.cc:1034] running on GCE node
W20260812 03:01:32.552417 24368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:33.746450 24343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:33.749177 24343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:33.750535 24343 hybrid_clock.cc:648] HybridClock initialized: now 1786503693750502 us; error 48 us; skew 500 ppm
I20260812 03:01:33.751398 24343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:33 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503693, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:33.760984 24343 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:01:33.766378 24343 webserver.cc:460] Webserver started at http://127.10.118.126:36145/ using document root <none> and password file <none>
I20260812 03:01:33.767355 24343 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:33.767607 24343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:33.775733 24343 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 03:01:33.780202 24388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:33.781378 24343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260812 03:01:33.781723 24343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "fe4a1fe1bce8473cbc4347baecdaabe2"
format_stamp: "Formatted at 2026-08-12 03:01:25 on dist-test-slave-zpvx"
server_key: "a66b9d77d5829ed83faf7896cb6ad1d6"
server_key_iv: "f750fd2b5717b3cdc313026a83741892"
server_key_version: "encryptionkey@0"
I20260812 03:01:33.784101 24343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:33.826057 24343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:33.831548 24343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:33.832029 24343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:33.901844 24343 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:41479
I20260812 03:01:33.902005 24487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:41479 every 8 connection(s)
I20260812 03:01:33.904860 24343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:01:33.908218 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 24343
I20260812 03:01:33.911439 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:32987
--local_ip_for_outbound_sockets=127.10.118.65
--tserver_master_addrs=127.10.118.126:41479
--webserver_port=46081
--webserver_interface=127.10.118.65
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.118.65
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 03:01:33.921840 24488 sys_catalog.cc:263] Verifying existing consensus state
I20260812 03:01:33.927623 24488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Bootstrap starting.
I20260812 03:01:33.966622 24488 log.cc:824] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Log is configured to *not* fsync() on all Append() calls
I20260812 03:01:33.987955 24488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 03:01:33.988965 24488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Bootstrap complete.
I20260812 03:01:34.019109 24488 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:34.021337 24488 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4a1fe1bce8473cbc4347baecdaabe2, State: Initialized, Role: FOLLOWER
I20260812 03:01:34.022184 24488 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:34.022686 24488 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:01:34.022989 24488 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:01:34.023313 24488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 1 FOLLOWER]: Advancing to term 2
I20260812 03:01:34.029500 24488 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:34.030247 24488 leader_election.cc:302] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4a1fe1bce8473cbc4347baecdaabe2; no voters:
I20260812 03:01:34.032471 24488 leader_election.cc:288] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 03:01:34.032886 24493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 03:01:34.036038 24493 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [term 2 LEADER]: Becoming Leader. State: Replica: fe4a1fe1bce8473cbc4347baecdaabe2, State: Running, Role: LEADER
I20260812 03:01:34.036867 24493 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } }
I20260812 03:01:34.037617 24488 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:01:34.048398 24500 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4a1fe1bce8473cbc4347baecdaabe2. Latest consensus state: current_term: 2 leader_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } } }
I20260812 03:01:34.050673 24500 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:34.049897 24497 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4a1fe1bce8473cbc4347baecdaabe2" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41479 } } }
I20260812 03:01:34.052814 24497 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:34.062865 24516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:01:34.077432 24516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:01:34.083520 24516 catalog_manager.cc:1259] Loaded cluster ID: 61630520c2f949b4854847c453de2602
I20260812 03:01:34.085155 24516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:01:34.095901 24516 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:01:34.101966 24516 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe4a1fe1bce8473cbc4347baecdaabe2: Loaded TSK: 0
I20260812 03:01:34.104360 24516 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 03:01:34.268815 24491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:34.269397 24491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:34.270056 24491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:34.300992 24491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:34.301376 24491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:34.302306 24491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:01:34.339840 24491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.65.keytab
--principal=oryx/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:32987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=46081
--tserver_master_addrs=127.10.118.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:34.341081 24491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:34.342134 24491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:34.363171 24548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:34.364437 24551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:34.363190 24547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:35.578272 24550 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1213 milliseconds
I20260812 03:01:35.578439 24491 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:01:35.579950 24491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:35.582739 24491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:35.584194 24491 hybrid_clock.cc:648] HybridClock initialized: now 1786503695584149 us; error 48 us; skew 500 ppm
I20260812 03:01:35.585289 24491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:35 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503695, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:35.592460 24491 init.cc:377] Logged in from keytab as oryx/127.10.118.65@KRBTEST.COM (short username oryx)
I20260812 03:01:35.598412 24491 webserver.cc:460] Webserver started at http://127.10.118.65:46081/ using document root <none> and password file <none>
I20260812 03:01:35.599328 24491 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:35.599565 24491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:35.608582 24491 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260812 03:01:35.613117 24561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:35.614338 24491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 03:01:35.614686 24491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "48f6355857bd4fcda480ed0fdb69ba4b"
format_stamp: "Formatted at 2026-08-12 03:01:27 on dist-test-slave-zpvx"
server_key: "f6e0ff903612311c796189adcf41dbed"
server_key_iv: "008fd4c32e89febe9aa94130c5226d2c"
server_key_version: "encryptionkey@0"
I20260812 03:01:35.617098 24491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:35.656975 24491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:35.663326 24491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:35.663831 24491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:35.667505 24491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:35.673142 24491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:35.673360 24491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:35.673720 24491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:35.673893 24491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:35.854012 24491 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:32987
I20260812 03:01:35.854193 24781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:32987 every 8 connection(s)
I20260812 03:01:35.856851 24491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:01:35.863978 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 24491
I20260812 03:01:35.866081 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:35977
--local_ip_for_outbound_sockets=127.10.118.66
--tserver_master_addrs=127.10.118.126:41479
--webserver_port=37479
--webserver_interface=127.10.118.66
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.118.66
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 03:01:35 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:35 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:35.904254 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:35.861788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:50717)
0812 03:01:35.874246 (+ 12458us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:35.874363 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:35.879432 (+ 5069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:35.881348 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:35.881389 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:35.882693 (+ 1304us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:35.884013 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:35.884096 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.887429 (+ 3333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.887467 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:35.890477 (+ 3010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:35.890541 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.892632 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.892689 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:35.893323 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:35.894587 (+ 1264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:35.894713 (+ 126us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:35.901868 (+ 7155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11854,"thread_start_us":11046,"threads_started":1}
W20260812 03:01:35.905860 24782 heartbeater.cc:644] Failed to heartbeat to 127.10.118.126:41479 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.118.126:41479: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
W20260812 03:01:35.921149 24785 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:35.861779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:50717 (local address 127.10.118.126:41479)
0812 03:01:35.864094 (+ 2315us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:35.864140 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:35.874608 (+ 10468us) server_negotiation.cc:407] Connection header received
0812 03:01:35.880149 (+ 5541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:35.880202 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:35.880700 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:35.881137 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:35.884456 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.887102 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.890828 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.892222 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.892603 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:35.919216 (+ 26613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:50717: BlockingRecv error: recv got EOF from 127.10.118.65:50717 (error 108)
Metrics: {"server-negotiator.queue_time_us":1709,"thread_start_us":2460,"threads_started":1}
Aug 12 03:01:35 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:35 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:35.933359 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:35.907098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:56937)
0812 03:01:35.912890 (+ 5792us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:35.912935 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:35.913184 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:35.916657 (+ 3473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:35.916698 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:35.917699 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:35.918306 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:35.918371 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.921783 (+ 3412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.921810 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:35.923610 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:35.923675 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.925670 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.925699 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:35.926051 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:35.926923 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:35.927004 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:35.931702 (+ 4698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5355,"thread_start_us":11987,"threads_started":1}
W20260812 03:01:35.933882 24790 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:35.907268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:56937 (local address 127.10.118.126:41479)
0812 03:01:35.913413 (+ 6145us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:35.913446 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:35.913510 (+ 64us) server_negotiation.cc:407] Connection header received
0812 03:01:35.913698 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:35.913737 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:35.914040 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:35.914337 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:35.919712 (+ 5375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.921457 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.923959 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.924927 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.925190 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:35.932277 (+ 7087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:56937: BlockingRecv error: recv got EOF from 127.10.118.65:56937 (error 108)
Metrics: {"server-negotiator.queue_time_us":5665,"thread_start_us":5360,"threads_started":1}
Aug 12 03:01:35 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:35 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:35.950651 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:35.935098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:36703)
0812 03:01:35.935583 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:35.935625 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:35.935844 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:35.936615 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:35.936644 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:35.937362 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:35.937770 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:35.937815 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.939870 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.939897 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:35.941670 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:35.941717 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.943168 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.943206 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:35.943689 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:35.944630 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:35.944725 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:35.949149 (+ 4424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260812 03:01:35.950814 24790 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:35.935328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:36703 (local address 127.10.118.126:41479)
0812 03:01:35.935806 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:35.935835 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:35.935878 (+ 43us) server_negotiation.cc:407] Connection header received
0812 03:01:35.936068 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:35.936093 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:35.936321 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:35.936520 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:35.938054 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.939572 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.941942 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:35.942841 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:35.943127 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:35.949580 (+ 6453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:36703: BlockingRecv error: recv got EOF from 127.10.118.65:36703 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260812 03:01:35.951596 24782 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 03:01:36.215636 24789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:36.216184 24789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:36.216862 24789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:36.247560 24789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:36.247958 24789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:36.248925 24789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:01:36.284075 24789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.66.keytab
--principal=oryx/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:35977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=37479
--tserver_master_addrs=127.10.118.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:36.285290 24789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:36.286319 24789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:36.306851 24803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 12 03:01:36 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:36 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:36.992952 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:36.953267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:41579)
0812 03:01:36.968836 (+ 15569us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:36.968917 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:36.969361 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:36.971609 (+ 2248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:36.971654 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:36.973019 (+ 1365us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:36.973761 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:36.973836 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:36.976464 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:36.976511 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:36.979531 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:36.979609 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:36.981717 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:36.981831 (+ 114us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:36.982440 (+ 609us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:36.983686 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:36.983813 (+ 127us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:36.990521 (+ 6708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14876,"thread_start_us":16204,"threads_started":1}
W20260812 03:01:36.993306 24808 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:36.953524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:41579 (local address 127.10.118.126:41479)
0812 03:01:36.970365 (+ 16841us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:36.970410 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:36.970484 (+ 74us) server_negotiation.cc:407] Connection header received
0812 03:01:36.970718 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:36.970758 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:36.971141 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:36.972316 (+ 1175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:36.974160 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:36.976059 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:36.979924 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:36.981273 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:36.981666 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:36.991182 (+ 9516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:41579: BlockingRecv error: recv got EOF from 127.10.118.65:41579 (error 108)
Metrics: {"server-negotiator.queue_time_us":15202,"thread_start_us":16469,"threads_started":1}
W20260812 03:01:36.309064 24797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:36.306903 24799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:37.552062 24800 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1243 milliseconds
I20260812 03:01:37.552192 24789 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:01:37.553570 24789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:37.556396 24789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:37.557826 24789 hybrid_clock.cc:648] HybridClock initialized: now 1786503697557765 us; error 82 us; skew 500 ppm
I20260812 03:01:37.558600 24789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:37 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503697, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:37.564710 24789 init.cc:377] Logged in from keytab as oryx/127.10.118.66@KRBTEST.COM (short username oryx)
I20260812 03:01:37.570079 24789 webserver.cc:460] Webserver started at http://127.10.118.66:37479/ using document root <none> and password file <none>
I20260812 03:01:37.571017 24789 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:37.571288 24789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:37.579519 24789 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 03:01:37.583796 24825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:37.584816 24789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 03:01:37.585143 24789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "f28b7849e7524cb39de614ef207d4609"
format_stamp: "Formatted at 2026-08-12 03:01:29 on dist-test-slave-zpvx"
server_key: "42832ac630c7754abd68da9c414b0d01"
server_key_iv: "07e2553c940433aef268c33e153e7b09"
server_key_version: "encryptionkey@0"
I20260812 03:01:37.587462 24789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:37.626220 24789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:37.631661 24789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:37.632118 24789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:37.634673 24789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:37.639140 24789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:37.639390 24789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:37.639706 24789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:37.639887 24789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:37.775893 24789 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:35977
I20260812 03:01:37.776075 25026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:35977 every 8 connection(s)
I20260812 03:01:37.778852 24789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:01:37.788442 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 24789
I20260812 03:01:37.790153 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:44801
--local_ip_for_outbound_sockets=127.10.118.67
--tserver_master_addrs=127.10.118.126:41479
--webserver_port=45535
--webserver_interface=127.10.118.67
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.118.67
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 03:01:37 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:37 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:37.811450 25030 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:37.783582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:44409 (local address 127.10.118.126:41479)
0812 03:01:37.784688 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:37.784705 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:37.787350 (+ 2645us) server_negotiation.cc:407] Connection header received
0812 03:01:37.792176 (+ 4826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:37.792212 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:37.792531 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:37.792749 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:37.795529 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.797181 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.800442 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.801123 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.801332 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:37.809997 (+ 8665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:44409: BlockingRecv error: recv got EOF from 127.10.118.66:44409 (error 108)
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":444,"threads_started":1}
I20260812 03:01:37.811956 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:37.783875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:44409)
0812 03:01:37.787111 (+ 3236us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:37.787177 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:37.791786 (+ 4609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:37.792958 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:37.792982 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:37.794056 (+ 1074us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:37.795173 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:37.795242 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.797467 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.797500 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:37.800093 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:37.800152 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.801474 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.801517 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:37.802131 (+ 614us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:37.803227 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:37.803374 (+ 147us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:37.809577 (+ 6203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2681,"thread_start_us":1844,"threads_started":1}
W20260812 03:01:37.812943 25027 heartbeater.cc:644] Failed to heartbeat to 127.10.118.126:41479 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.118.126:41479: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Aug 12 03:01:37 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:37 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:37.826720 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:37.814119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:45579)
0812 03:01:37.814655 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:37.814699 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:37.814960 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:37.815631 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:37.815658 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:37.816226 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:37.816580 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:37.816612 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.817924 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.817951 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:37.819042 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:37.819079 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.820150 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.820181 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:37.820562 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:37.821292 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:37.821357 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:37.825143 (+ 3786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260812 03:01:37.826822 25030 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:37.814178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:45579 (local address 127.10.118.126:41479)
0812 03:01:37.814694 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:37.814722 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:37.814825 (+ 103us) server_negotiation.cc:407] Connection header received
0812 03:01:37.815184 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:37.815205 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:37.815377 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:37.815545 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:37.816833 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.817708 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.819317 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.819881 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.820157 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:37.825541 (+ 5384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:45579: BlockingRecv error: recv got EOF from 127.10.118.66:45579 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Aug 12 03:01:37 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:37 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:37.840121 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:37.828188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:38711)
0812 03:01:37.828685 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:37.828729 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:37.828936 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:37.829589 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:37.829622 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:37.830165 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:37.830446 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:37.830474 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.831764 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.831786 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:37.832936 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:37.832970 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.834010 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.834040 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:37.834404 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:37.835124 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:37.835183 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:37.838892 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260812 03:01:37.840822 25027 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 03:01:37.840802 25030 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:37.828387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:38711 (local address 127.10.118.126:41479)
0812 03:01:37.828781 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:37.828798 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:37.828828 (+ 30us) server_negotiation.cc:407] Connection header received
0812 03:01:37.829146 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:37.829164 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:37.829343 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:37.829509 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:37.830689 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.831537 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.833175 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:37.833749 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:37.834021 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:37.839488 (+ 5467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:38711: BlockingRecv error: recv got EOF from 127.10.118.66:38711 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 12 03:01:38 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:38 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:38.010198 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:37.995399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:33575)
0812 03:01:37.997151 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:37.997189 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:37.997467 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:37.998147 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:37.998173 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:37.998847 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:37.999175 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:37.999210 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.000640 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.000662 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:38.001973 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:38.002012 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.003085 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.003126 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:38.003498 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:38.004481 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:38.004544 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:38.008421 (+ 3877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1426,"thread_start_us":1171,"threads_started":1}
W20260812 03:01:38.010183 25030 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:37.995736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:33575 (local address 127.10.118.126:41479)
0812 03:01:37.996534 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:37.996561 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:37.997322 (+ 761us) server_negotiation.cc:407] Connection header received
0812 03:01:37.997699 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:37.997721 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:37.997890 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:37.998069 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:37.999430 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.000349 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.002232 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.002814 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.003133 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:38.008854 (+ 5721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:33575: BlockingRecv error: recv got EOF from 127.10.118.65:33575 (error 108)
Metrics: {"server-negotiator.queue_time_us":204}
W20260812 03:01:38.112287 25036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:38.112882 25036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:38.113523 25036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:38.144508 25036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:38.144870 25036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:38.145884 25036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:01:38.180020 25036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/krb5kdc/oryx_127.10.118.67.keytab
--principal=oryx/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:44801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=45535
--tserver_master_addrs=127.10.118.126:41479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:38.181234 25036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:38.182286 25036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:38.200116 25064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 12 03:01:38 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:38 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:38.862598 25070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:38.842179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:35783)
0812 03:01:38.844269 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:38.844307 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:38.844622 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:38.845320 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:38.845347 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:38.846064 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:38.846409 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:38.846443 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.847818 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.847839 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:38.849160 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:38.849197 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.850435 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.850465 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:38.850827 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:38.856941 (+ 6114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:38.857003 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:38.860780 (+ 3777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1734,"thread_start_us":1461,"threads_started":1}
W20260812 03:01:38.862623 25069 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:38.842154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:35783 (local address 127.10.118.126:41479)
0812 03:01:38.843658 (+ 1504us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:38.843683 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:38.844459 (+ 776us) server_negotiation.cc:407] Connection header received
0812 03:01:38.844869 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:38.844888 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:38.845055 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:38.845242 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:38.846672 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.847583 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.849414 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:38.850110 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:38.850400 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:38.861195 (+ 10795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:35783: BlockingRecv error: recv got EOF from 127.10.118.66:35783 (error 108)
Metrics: {"server-negotiator.queue_time_us":958,"thread_start_us":650,"threads_started":1}
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:39.027166 25069 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:39.012434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:34403 (local address 127.10.118.126:41479)
0812 03:01:39.012853 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:39.012893 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:39.014622 (+ 1729us) server_negotiation.cc:407] Connection header received
0812 03:01:39.015013 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:39.015040 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:39.015208 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:39.015384 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:39.016731 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.017637 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.019281 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.019865 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.020141 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.025861 (+ 5720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:34403: BlockingRecv error: recv got EOF from 127.10.118.65:34403 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 03:01:39.027163 25071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:39.012424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:34403)
0812 03:01:39.014445 (+ 2021us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:39.014484 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:39.014783 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:39.015472 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:39.015500 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:39.016142 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:39.016472 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.016505 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.017870 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.017899 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.019027 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.019063 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.020134 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.020169 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.020573 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.021348 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:39.021407 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:39.025457 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1685,"thread_start_us":1396,"threads_started":1}
W20260812 03:01:38.200193 25053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:38.201998 25036 server_base.cc:1034] running on GCE node
W20260812 03:01:38.200186 25054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:39.393322 25036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:39.395634 25036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:39.397014 25036 hybrid_clock.cc:648] HybridClock initialized: now 1786503699396983 us; error 43 us; skew 500 ppm
I20260812 03:01:39.397855 25036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503699, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:01:39.404162 25036 init.cc:377] Logged in from keytab as oryx/127.10.118.67@KRBTEST.COM (short username oryx)
I20260812 03:01:39.409430 25036 webserver.cc:460] Webserver started at http://127.10.118.67:45535/ using document root <none> and password file <none>
I20260812 03:01:39.410365 25036 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:39.410619 25036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:39.418676 25036 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 03:01:39.422890 25081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:39.423943 25036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 03:01:39.424273 25036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "24d281942e1b48a1a84a03bd6e55b3ee"
format_stamp: "Formatted at 2026-08-12 03:01:31 on dist-test-slave-zpvx"
server_key: "07c53cc019a7ca3f66299eb8c8289a76"
server_key_iv: "4e0d66c27f82f60382842701e60f025a"
server_key_version: "encryptionkey@0"
I20260812 03:01:39.426631 25036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:39.471284 25036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:39.476984 25036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:39.477514 25036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:39.480113 25036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:39.484028 25036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:39.484268 25036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:39.484544 25036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:39.484728 25036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:39.623057 25036 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:44801
I20260812 03:01:39.623234 25297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:44801 every 8 connection(s)
I20260812 03:01:39.625877 25036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:01:39.631459 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 25036
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:39.658229 25301 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:39.630226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:43593 (local address 127.10.118.126:41479)
0812 03:01:39.631489 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:39.631515 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:39.634101 (+ 2586us) server_negotiation.cc:407] Connection header received
0812 03:01:39.641328 (+ 7227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:39.641360 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:39.641535 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:39.641715 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:39.643969 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.645474 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.648823 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.649511 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.649823 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.656682 (+ 6859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:43593: BlockingRecv error: recv got EOF from 127.10.118.67:43593 (error 108)
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":794,"thread_start_us":511,"threads_started":1}
I20260812 03:01:39.658620 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:39.630599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:43593)
0812 03:01:39.633855 (+ 3256us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:39.633923 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:39.640962 (+ 7039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:39.641991 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:39.642027 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:39.642843 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:39.643687 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.643729 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.645790 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.645818 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.648541 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.648595 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.649846 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.649883 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.650352 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.651152 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:39.651214 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:39.656271 (+ 5057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2641,"thread_start_us":1790,"threads_started":1}
W20260812 03:01:39.659637 25298 heartbeater.cc:644] Failed to heartbeat to 127.10.118.126:41479 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.118.126:41479: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
I20260812 03:01:39.660495 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:39.634280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.10.118.126:41479)
0812 03:01:39.635327 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:39.635342 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:39.635542 (+ 200us) server_negotiation.cc:407] Connection header received
0812 03:01:39.636051 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:39.636068 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:39.636240 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:39.636437 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:39.640359 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.641384 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.643533 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.644147 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.644507 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.649288 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:39.649312 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:39.649341 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:01:39.649444 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:39.653836 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:39.654965 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:39.654988 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:39.655008 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:39.655257 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:01:39.655914 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:01:39.655934 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:01:39.655947 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:01:39.656670 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:39.657150 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:39.657866 (+ 716us) server_negotiation.cc:299] Negotiation successful
0812 03:01:39.658461 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":739,"thread_start_us":588,"threads_started":1}
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:39.672487 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:39.660752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:40601)
0812 03:01:39.661397 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:39.661441 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:39.661678 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:39.662294 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:39.662320 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:39.662918 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:39.663279 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.663308 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.664847 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.664897 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.666129 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.666163 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.667190 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.667222 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.667571 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.668194 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:39.668251 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:39.671494 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200}
W20260812 03:01:39.672652 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:39.661027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:40601 (local address 127.10.118.126:41479)
0812 03:01:39.661394 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:39.661409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:39.661554 (+ 145us) server_negotiation.cc:407] Connection header received
0812 03:01:39.661859 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:39.661876 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:39.662029 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:39.662225 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:39.663531 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.664645 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.666360 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.666925 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.667206 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.671796 (+ 4590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:40601: BlockingRecv error: recv got EOF from 127.10.118.67:40601 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:39.685500 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:39.673751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:52069)
0812 03:01:39.674226 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:39.674269 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:39.674480 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:39.675112 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:39.675138 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:39.675695 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:39.675966 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.675997 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.677417 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.677447 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.678697 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.678734 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.679761 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.679792 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.680142 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.680769 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:39.680826 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:39.683921 (+ 3095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 03:01:39.685655 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:39.673887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:52069 (local address 127.10.118.126:41479)
0812 03:01:39.674410 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:39.674433 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:39.674476 (+ 43us) server_negotiation.cc:407] Connection header received
0812 03:01:39.674707 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:39.674729 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:39.674886 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:39.675042 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:39.676210 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.677208 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.678928 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.679494 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.679773 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.684304 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:52069: BlockingRecv error: recv got EOF from 127.10.118.67:52069 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 03:01:39.686146 25298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:39 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:39.878568 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:39.864566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:41823)
0812 03:01:39.866638 (+ 2072us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:39.866690 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:39.866910 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:39.867502 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:39.867520 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:39.868161 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:39.868555 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.868593 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.869919 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.869939 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.871103 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:39.871171 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.872236 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.872259 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:39.872680 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.873383 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:39.873442 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:39.877446 (+ 4004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":1277,"threads_started":1}
W20260812 03:01:39.878706 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:39.864696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:41823 (local address 127.10.118.126:41479)
0812 03:01:39.865312 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:39.865344 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:39.866801 (+ 1457us) server_negotiation.cc:407] Connection header received
0812 03:01:39.867085 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:39.867103 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:39.867285 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:39.867417 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:39.868789 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.869714 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.871389 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:39.871971 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:39.872179 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:39.877858 (+ 5679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:41823: BlockingRecv error: recv got EOF from 127.10.118.66:41823 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 12 03:01:40 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:40 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:40.042943 25309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:40.028862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:34321)
0812 03:01:40.030441 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:40.030478 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:40.030748 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:40.031407 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:40.031436 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:40.032095 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:40.032496 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:40.032533 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.033902 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.033923 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:40.035644 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:40.035694 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.037018 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.037044 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:40.037301 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:40.038017 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:40.038076 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:40.041872 (+ 3796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1251,"thread_start_us":1003,"threads_started":1}
W20260812 03:01:40.043037 25306 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:40.029145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:34321 (local address 127.10.118.126:41479)
0812 03:01:40.029815 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:40.029841 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:40.030601 (+ 760us) server_negotiation.cc:407] Connection header received
0812 03:01:40.030964 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:40.030983 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:40.031156 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:40.031321 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:40.032748 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.033659 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.035926 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.036790 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.036991 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:40.042215 (+ 5224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:34321: BlockingRecv error: recv got EOF from 127.10.118.65:34321 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 12 03:01:40 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:40 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:40.703253 25317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:40.687543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:33089 (local address 127.10.118.126:41479)
0812 03:01:40.689268 (+ 1725us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:40.689290 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:40.690031 (+ 741us) server_negotiation.cc:407] Connection header received
0812 03:01:40.690559 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:40.690586 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:40.690805 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:40.690986 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:40.693175 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.694249 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.695978 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.696634 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.696842 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:40.701812 (+ 4970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:33089: BlockingRecv error: recv got EOF from 127.10.118.67:33089 (error 108)
Metrics: {"server-negotiator.queue_time_us":1289,"thread_start_us":903,"threads_started":1}
I20260812 03:01:40.703236 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:40.687379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:33089)
0812 03:01:40.689849 (+ 2470us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:40.689901 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:40.690246 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:40.691139 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:40.691196 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:40.692290 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:40.692854 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:40.692908 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.694441 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.694461 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:40.695765 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:40.695805 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.696899 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.696927 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:40.697170 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:40.697854 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:40.697913 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:40.701409 (+ 3496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1955,"thread_start_us":1672,"threads_started":1}
Aug 12 03:01:40 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:40 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:40.894235 25317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:40.880209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:40123 (local address 127.10.118.126:41479)
0812 03:01:40.880629 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:40.880647 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:40.882231 (+ 1584us) server_negotiation.cc:407] Connection header received
0812 03:01:40.882504 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:40.882526 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:40.882688 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:40.882827 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:40.884069 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.885025 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.886603 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.887160 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.887369 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:40.892954 (+ 5585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:40123: BlockingRecv error: recv got EOF from 127.10.118.66:40123 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 03:01:40.894299 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:40.880073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:40123)
0812 03:01:40.882080 (+ 2007us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:40.882120 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:40.882334 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:40.882919 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:40.882939 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:40.883549 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:40.883858 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:40.883892 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.885229 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.885250 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:40.886389 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:40.886430 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:40.887411 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:40.887434 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:40.887695 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:40.888399 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:40.888460 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:40.892559 (+ 4099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1666,"thread_start_us":1395,"threads_started":1}
Aug 12 03:01:41 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:41 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:41.058525 25317 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:41.044610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:34203 (local address 127.10.118.126:41479)
0812 03:01:41.044993 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:41.045011 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:41.046609 (+ 1598us) server_negotiation.cc:407] Connection header received
0812 03:01:41.046906 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:41.046948 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:41.047147 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:41.047276 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:41.048542 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.049478 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.051117 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.051695 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.051972 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:41.057235 (+ 5263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:34203: BlockingRecv error: recv got EOF from 127.10.118.65:34203 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 03:01:41.058537 25323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:41.044500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:34203)
0812 03:01:41.046466 (+ 1966us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:41.046507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:41.046706 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:41.047367 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:41.047387 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:41.047975 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:41.048291 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:41.048325 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.049678 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.049699 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:41.050860 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:41.050898 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.051968 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.052004 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:41.052420 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:41.053200 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:41.053262 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:41.056820 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":1359,"threads_started":1}
Aug 12 03:01:41 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:41 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:41.719517 25324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:41.705099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:41811)
0812 03:01:41.706844 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:41.706882 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:41.707146 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:41.708353 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:41.708415 (+ 62us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:41.709217 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:41.709545 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:41.709597 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.710911 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.710932 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:41.712104 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:41.712144 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.713253 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.713277 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:41.713523 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:41.714271 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:41.714330 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:41.718351 (+ 4021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1415,"thread_start_us":1191,"threads_started":1}
W20260812 03:01:41.720072 25325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:41.705156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:41811 (local address 127.10.118.126:41479)
0812 03:01:41.707693 (+ 2537us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:41.707707 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:41.707741 (+ 34us) server_negotiation.cc:407] Connection header received
0812 03:01:41.707844 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:41.707859 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:41.708024 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:41.708159 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:41.709803 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.710698 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.712405 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.713008 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.713213 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:41.718838 (+ 5625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:41811: BlockingRecv error: recv got EOF from 127.10.118.67:41811 (error 108)
Metrics: {"server-negotiator.queue_time_us":2234,"thread_start_us":1953,"threads_started":1}
Aug 12 03:01:41 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:41 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:41.912413 25325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:41.896008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:37257 (local address 127.10.118.126:41479)
0812 03:01:41.896655 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:41.896694 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:41.897671 (+ 977us) server_negotiation.cc:407] Connection header received
0812 03:01:41.897942 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:41.897961 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:41.898134 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:41.898275 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:41.899797 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.901185 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.903713 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.904957 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.905229 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:41.911438 (+ 6209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:37257: BlockingRecv error: recv got EOF from 127.10.118.66:37257 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 03:01:41.912755 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:41.895886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:37257)
0812 03:01:41.897519 (+ 1633us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:41.897560 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:41.897768 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:41.898393 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:41.898416 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:41.899076 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:41.899508 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:41.899550 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.901433 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.901465 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:41.903374 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:41.903439 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:41.905267 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:41.905306 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:41.905694 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:41.906525 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:41.906595 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:41.911075 (+ 4480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1335,"thread_start_us":1055,"threads_started":1}
Aug 12 03:01:42 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:42 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:42.074437 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:42.060246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:52721)
0812 03:01:42.062022 (+ 1776us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:42.062074 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:42.062352 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:42.063080 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:42.063099 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:42.063863 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:42.064188 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:42.064223 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.065715 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.065739 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:42.067059 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:42.067099 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.068087 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.068111 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:42.068404 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:42.069129 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:42.069210 (+ 81us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:42.072736 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1364,"thread_start_us":1067,"threads_started":1}
W20260812 03:01:42.074431 25325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:42.060472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:52721 (local address 127.10.118.126:41479)
0812 03:01:42.061129 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:42.061165 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:42.062229 (+ 1064us) server_negotiation.cc:407] Connection header received
0812 03:01:42.062581 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:42.062610 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:42.062805 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:42.063002 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:42.064498 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.065530 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.067289 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.067863 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.068068 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:42.073163 (+ 5095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:52721: BlockingRecv error: recv got EOF from 127.10.118.65:52721 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 03:01:42 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:42 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:42.735574 25331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:42.721097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:47155)
0812 03:01:42.722814 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:42.722853 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:42.723042 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:42.724444 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:42.724463 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:42.725040 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:42.725349 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:42.725381 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.727035 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.727056 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:42.728618 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:42.728659 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.730153 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.730177 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:42.730417 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:42.731091 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:42.731150 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:42.734513 (+ 3363us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1384,"thread_start_us":1161,"threads_started":1}
W20260812 03:01:42.736164 25332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:42.721415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:47155 (local address 127.10.118.126:41479)
0812 03:01:42.723729 (+ 2314us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:42.723749 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:42.723801 (+ 52us) server_negotiation.cc:407] Connection header received
0812 03:01:42.723960 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:42.723975 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:42.724170 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:42.724328 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:42.725592 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.726793 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.728864 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.729931 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.730138 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:42.734893 (+ 4755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:47155: BlockingRecv error: recv got EOF from 127.10.118.67:47155 (error 108)
Metrics: {"server-negotiator.queue_time_us":1919,"thread_start_us":1550,"threads_started":1}
Aug 12 03:01:42 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:42 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:42.928758 25334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:42.914380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:54759)
0812 03:01:42.916076 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:42.916116 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:42.916418 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:42.917129 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:42.917158 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:42.917801 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:42.918121 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:42.918156 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.919600 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.919633 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:42.920867 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:42.920906 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.922181 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.922216 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:42.922599 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:42.923741 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:42.923802 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:42.927631 (+ 3829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1361,"thread_start_us":1110,"threads_started":1}
W20260812 03:01:42.928833 25332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:42.914773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:54759 (local address 127.10.118.126:41479)
0812 03:01:42.915442 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:42.915475 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:42.916235 (+ 760us) server_negotiation.cc:407] Connection header received
0812 03:01:42.916665 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:42.916686 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:42.916867 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:42.917030 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:42.918386 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.919358 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.921124 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:42.921889 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:42.922201 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:42.927959 (+ 5758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:54759: BlockingRecv error: recv got EOF from 127.10.118.66:54759 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Aug 12 03:01:43 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:43 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:43.089823 25338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:43.076013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:45395)
0812 03:01:43.077595 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:43.077642 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:43.077911 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:43.078615 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:43.078645 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:43.079297 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:43.079652 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:43.079688 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.081313 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.081341 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:43.083008 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:43.083047 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.084033 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.084057 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:43.084307 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:43.085002 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:43.085062 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:43.088735 (+ 3673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":993,"threads_started":1}
W20260812 03:01:43.089901 25332 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:43.076105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:45395 (local address 127.10.118.126:41479)
0812 03:01:43.076732 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:43.076768 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:43.077762 (+ 994us) server_negotiation.cc:407] Connection header received
0812 03:01:43.078159 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:43.078182 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:43.078350 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:43.078516 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:43.079910 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.081028 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.083238 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.083806 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.084002 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:43.089065 (+ 5063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:45395: BlockingRecv error: recv got EOF from 127.10.118.65:45395 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 12 03:01:43 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:43 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:43.751694 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:43.737248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:56183)
0812 03:01:43.739735 (+ 2487us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:43.739788 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:43.740069 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:43.740760 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:43.740779 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:43.741364 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:43.741689 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:43.741735 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.743077 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.743097 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:43.744264 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:43.744301 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.745390 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.745414 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:43.745740 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:43.746495 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:43.746583 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:43.750251 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2049,"thread_start_us":1691,"threads_started":1}
W20260812 03:01:43.751713 25340 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:43.737482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:56183 (local address 127.10.118.126:41479)
0812 03:01:43.739304 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:43.739325 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:43.739912 (+ 587us) server_negotiation.cc:407] Connection header received
0812 03:01:43.740293 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:43.740312 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:43.740530 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:43.740674 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:43.741952 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.742885 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.744522 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.745170 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.745368 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:43.750519 (+ 5151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:56183: BlockingRecv error: recv got EOF from 127.10.118.67:56183 (error 108)
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":877,"threads_started":1}
Aug 12 03:01:43 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:43 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:43.952919 25340 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:43.930517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:58845 (local address 127.10.118.126:41479)
0812 03:01:43.931237 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:43.931263 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:43.932079 (+ 816us) server_negotiation.cc:407] Connection header received
0812 03:01:43.932584 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:43.932605 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:43.932778 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:43.932946 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:43.934687 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.935657 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.937371 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.938049 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.938333 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:43.951601 (+ 13268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:58845: BlockingRecv error: recv got EOF from 127.10.118.66:58845 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 03:01:43.952941 25342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:43.930363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:58845)
0812 03:01:43.931916 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:43.931955 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:43.932270 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:43.933045 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:43.933065 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:43.933817 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:43.934330 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:43.934384 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.935909 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.935931 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:43.937120 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:43.937158 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:43.938348 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:43.938384 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:43.938794 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:43.947308 (+ 8514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:43.947370 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:43.951157 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1235,"thread_start_us":975,"threads_started":1}
Aug 12 03:01:44 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:44 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:44.105763 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:44.091376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:43673)
0812 03:01:44.093571 (+ 2195us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:44.093612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:44.093830 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:44.094525 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:44.094546 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:44.095161 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:44.095490 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:44.095527 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.096906 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.096927 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:44.098096 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:44.098139 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.099169 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.099194 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:44.099467 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:44.100242 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:44.100327 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:44.104651 (+ 4324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1841,"thread_start_us":1590,"threads_started":1}
W20260812 03:01:44.105789 25340 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:44.091536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:43673 (local address 127.10.118.126:41479)
0812 03:01:44.091929 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:44.091947 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:44.093747 (+ 1800us) server_negotiation.cc:407] Connection header received
0812 03:01:44.094051 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:44.094079 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:44.094256 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:44.094395 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:44.095753 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.096711 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.098348 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.098921 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.099119 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:44.104982 (+ 5863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:43673: BlockingRecv error: recv got EOF from 127.10.118.65:43673 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 03:01:44 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:44 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:44.767864 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:44.753505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:36729)
0812 03:01:44.755246 (+ 1741us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:44.755286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:44.755472 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:44.757165 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:44.757184 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:44.757928 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:44.758330 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:44.758380 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.759819 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.759841 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:44.760996 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:44.761033 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.762134 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.762165 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:44.762527 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:44.763213 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:44.763274 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:44.766784 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1419,"thread_start_us":1175,"threads_started":1}
W20260812 03:01:44.768036 25348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:44.753669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:36729 (local address 127.10.118.126:41479)
0812 03:01:44.756167 (+ 2498us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:44.756188 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:44.756237 (+ 49us) server_negotiation.cc:407] Connection header received
0812 03:01:44.756447 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:44.756465 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:44.756656 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:44.756852 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:44.758611 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.759586 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.761240 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.761850 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.762124 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:44.767098 (+ 4974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:36729: BlockingRecv error: recv got EOF from 127.10.118.67:36729 (error 108)
Metrics: {"server-negotiator.queue_time_us":2142,"thread_start_us":1855,"threads_started":1}
Aug 12 03:01:44 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:44 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:44.967892 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:44.954478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:57607)
0812 03:01:44.956055 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:44.956095 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:44.956321 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:44.956915 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:44.956934 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:44.957582 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:44.957895 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:44.957929 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.959195 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.959220 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:44.960363 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:44.960421 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.961553 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.961585 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:44.961904 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:44.962779 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:44.962840 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:44.966833 (+ 3993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1268,"thread_start_us":1042,"threads_started":1}
W20260812 03:01:44.967967 25348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:44.954604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:57607 (local address 127.10.118.126:41479)
0812 03:01:44.955228 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:44.955259 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:44.956201 (+ 942us) server_negotiation.cc:407] Connection header received
0812 03:01:44.956502 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:44.956521 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:44.956710 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:44.956838 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:44.958112 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.959013 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.960591 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:44.961277 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:44.961527 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:44.967147 (+ 5620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:57607: BlockingRecv error: recv got EOF from 127.10.118.66:57607 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 12 03:01:45 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:45 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:45.121999 25348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:45.107808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:56611 (local address 127.10.118.126:41479)
0812 03:01:45.108515 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:45.108538 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:45.109136 (+ 598us) server_negotiation.cc:407] Connection header received
0812 03:01:45.109526 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:45.109544 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:45.109711 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:45.109841 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:45.111547 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.112544 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.114333 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.114896 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.115171 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:45.121182 (+ 6011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:56611: BlockingRecv error: recv got EOF from 127.10.118.65:56611 (error 108)
Metrics: {"server-negotiator.queue_time_us":177}
I20260812 03:01:45.122128 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:45.107703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:56611)
0812 03:01:45.108957 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:45.109003 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:45.109357 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:45.109936 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:45.109955 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:45.110778 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:45.111310 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:45.111358 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.112744 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.112776 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:45.114079 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:45.114116 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.115152 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.115197 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:45.115508 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:45.116454 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:45.116545 (+ 91us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:45.120890 (+ 4345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":471,"threads_started":1}
Aug 12 03:01:45 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:45 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:45.785794 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:45.769552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:35271)
0812 03:01:45.771551 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:45.771590 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:45.771798 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:45.772457 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:45.772477 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:45.773298 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:45.773751 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:45.773799 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.775473 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.775510 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:45.777377 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:45.777428 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.778552 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.778591 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:45.778973 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:45.779788 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:45.779868 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:45.784022 (+ 4154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1653,"thread_start_us":1402,"threads_started":1}
W20260812 03:01:45.785915 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:45.769667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:35271 (local address 127.10.118.126:41479)
0812 03:01:45.770810 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:45.770824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:45.771697 (+ 873us) server_negotiation.cc:407] Connection header received
0812 03:01:45.771968 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:45.771985 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:45.772156 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:45.772336 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:45.774046 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.775256 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.777654 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.778261 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.778461 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:45.784477 (+ 6016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:35271: BlockingRecv error: recv got EOF from 127.10.118.67:35271 (error 108)
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":603,"threads_started":1}
Aug 12 03:01:45 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:45 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:45.983553 25356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:45.969692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:49289)
0812 03:01:45.971319 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:45.971356 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:45.971623 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:45.972284 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:45.972312 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:45.972993 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:45.973303 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:45.973338 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.974867 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.974895 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:45.976050 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:45.976095 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.977216 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.977253 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:45.977611 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:45.978332 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:45.978391 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:45.982441 (+ 4050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1303,"thread_start_us":1016,"threads_started":1}
W20260812 03:01:45.983598 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:45.969878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:49289 (local address 127.10.118.126:41479)
0812 03:01:45.970646 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:45.970675 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:45.971484 (+ 809us) server_negotiation.cc:407] Connection header received
0812 03:01:45.971843 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:45.971863 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:45.972030 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:45.972211 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:45.973558 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.974648 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.976307 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:45.976949 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:45.977234 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:45.982757 (+ 5523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:49289: BlockingRecv error: recv got EOF from 127.10.118.66:49289 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Aug 12 03:01:46 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:46 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:46.136914 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:46.124058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:55641)
0812 03:01:46.125349 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:46.125387 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:46.125579 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:46.126139 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:46.126191 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:46.126753 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:46.127059 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:46.127092 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.128397 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.128417 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:46.129593 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:46.129629 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.130598 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.130621 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:46.130863 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:46.131581 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:46.131641 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:46.135412 (+ 3771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":640,"threads_started":1}
W20260812 03:01:46.136984 25355 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:46.124200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:55641 (local address 127.10.118.126:41479)
0812 03:01:46.124637 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:46.124656 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:46.125478 (+ 822us) server_negotiation.cc:407] Connection header received
0812 03:01:46.125745 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:46.125765 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:46.125929 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:46.126060 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:46.127295 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.128175 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.129803 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.130381 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.130573 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:46.135836 (+ 5263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:55641: BlockingRecv error: recv got EOF from 127.10.118.65:55641 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 12 03:01:46 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:46 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:46.801496 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:46.787607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:47967)
0812 03:01:46.789766 (+ 2159us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:46.789806 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:46.790079 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:46.790783 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:46.790809 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:46.791460 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:46.791871 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:46.791925 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.793378 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.793405 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:46.794574 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:46.794610 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.795682 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.795722 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:46.796088 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:46.796848 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:46.796908 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:46.800434 (+ 3526us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":1493,"threads_started":1}
W20260812 03:01:46.801664 25361 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:46.787716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:47967 (local address 127.10.118.126:41479)
0812 03:01:46.789550 (+ 1834us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:46.789574 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:46.789929 (+ 355us) server_negotiation.cc:407] Connection header received
0812 03:01:46.790314 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:46.790333 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:46.790498 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:46.790669 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:46.792140 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.793154 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.794828 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.795407 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.795697 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:46.800756 (+ 5059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:47967: BlockingRecv error: recv got EOF from 127.10.118.67:47967 (error 108)
Metrics: {"server-negotiator.queue_time_us":1380,"thread_start_us":1077,"threads_started":1}
Aug 12 03:01:46 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:46 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:47.001240 25361 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:46.985756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:47065 (local address 127.10.118.126:41479)
0812 03:01:46.986367 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:46.986414 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:46.987642 (+ 1228us) server_negotiation.cc:407] Connection header received
0812 03:01:46.988092 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:46.988115 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:46.988328 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:46.988503 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:46.990145 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.991075 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.993028 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.993739 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.993953 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:47.000255 (+ 6302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:47065: BlockingRecv error: recv got EOF from 127.10.118.66:47065 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 03:01:47.001221 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:46.985703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:47065)
0812 03:01:46.987463 (+ 1760us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:46.987509 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:46.987831 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:46.988597 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:46.988616 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:46.989428 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:46.989884 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:46.989934 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.991303 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.991334 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:46.992789 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:46.992832 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:46.993985 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:46.994009 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:46.994291 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:46.995266 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:46.995332 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:46.999892 (+ 4560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":1065,"threads_started":1}
Aug 12 03:01:47 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:47 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:47.152896 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:47.138626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:40221)
0812 03:01:47.139753 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:47.139804 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:47.140009 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:47.140832 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:47.140852 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:47.141705 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:47.142111 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:47.142164 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.143825 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.143854 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:47.145173 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:47.145210 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.146271 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.146301 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:47.146562 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:47.147416 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:47.147485 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:47.151119 (+ 3634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":452,"threads_started":1}
W20260812 03:01:47.152920 25361 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:47.138743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:40221 (local address 127.10.118.126:41479)
0812 03:01:47.139376 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:47.139404 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:47.139908 (+ 504us) server_negotiation.cc:407] Connection header received
0812 03:01:47.140229 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:47.140256 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:47.140517 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:47.140717 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:47.142418 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.143612 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.145407 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.145999 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.146303 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:47.151562 (+ 5259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:40221: BlockingRecv error: recv got EOF from 127.10.118.65:40221 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Aug 12 03:01:47 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:47 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:47.817405 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:47.803108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:51623)
0812 03:01:47.805206 (+ 2098us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:47.805244 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:47.805453 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:47.806024 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:47.806042 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:47.806652 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:47.807088 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:47.807125 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.808576 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.808596 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:47.809777 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:47.809816 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.810767 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.810792 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:47.811028 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:47.811783 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:47.811843 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:47.815917 (+ 4074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1752,"thread_start_us":1497,"threads_started":1}
W20260812 03:01:47.817615 25372 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:47.803223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:51623 (local address 127.10.118.126:41479)
0812 03:01:47.804493 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:47.804507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:47.805381 (+ 874us) server_negotiation.cc:407] Connection header received
0812 03:01:47.805620 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:47.805638 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:47.805812 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:47.805944 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:47.807319 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.808384 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.809988 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:47.810556 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:47.810747 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:47.816215 (+ 5468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:51623: BlockingRecv error: recv got EOF from 127.10.118.67:51623 (error 108)
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":579,"threads_started":1}
Aug 12 03:01:48 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:48 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:48.015781 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:48.002908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:60129)
0812 03:01:48.004228 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:48.004285 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:48.004548 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:48.005135 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:48.005152 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:48.005741 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:48.006055 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:48.006087 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.007694 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.007717 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:48.009178 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:48.009224 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.010220 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.010243 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:48.010478 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:48.011232 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:48.011292 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:48.014732 (+ 3440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":659,"threads_started":1}
W20260812 03:01:48.015872 25372 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:48.003042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:60129 (local address 127.10.118.126:41479)
0812 03:01:48.003461 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:48.003495 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:48.004398 (+ 903us) server_negotiation.cc:407] Connection header received
0812 03:01:48.004735 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:48.004754 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:48.004930 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:48.005053 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:48.006288 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.007486 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.009412 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.009987 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.010168 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:48.015049 (+ 4881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:60129: BlockingRecv error: recv got EOF from 127.10.118.66:60129 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 12 03:01:48 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:48 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:48.168159 25372 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:48.154667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:59869 (local address 127.10.118.126:41479)
0812 03:01:48.155077 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:48.155092 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:48.155811 (+ 719us) server_negotiation.cc:407] Connection header received
0812 03:01:48.156065 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:48.156082 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:48.156265 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:48.156414 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:48.157789 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.158731 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.160798 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.161568 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.161859 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:48.166881 (+ 5022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:59869: BlockingRecv error: recv got EOF from 127.10.118.65:59869 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 03:01:48.168198 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:48.154458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:59869)
0812 03:01:48.155657 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:48.155696 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:48.155899 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:48.156526 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:48.156545 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:48.157203 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:48.157533 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:48.157567 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.158939 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.158966 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:48.160545 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:48.160596 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.161867 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.161906 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:48.162179 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:48.162892 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:48.162953 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:48.166480 (+ 3527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":601,"threads_started":1}
Aug 12 03:01:48 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:48 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:48.834506 25379 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:48.819381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:44501 (local address 127.10.118.126:41479)
0812 03:01:48.820810 (+ 1429us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:48.820835 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:48.821521 (+ 686us) server_negotiation.cc:407] Connection header received
0812 03:01:48.821979 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:48.821998 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:48.822169 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:48.822339 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:48.824083 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.825103 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.827483 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.828085 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.828293 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:48.833606 (+ 5313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:44501: BlockingRecv error: recv got EOF from 127.10.118.67:44501 (error 108)
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":567,"threads_started":1}
I20260812 03:01:48.834546 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:48.819381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:44501)
0812 03:01:48.821315 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:48.821370 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:48.821748 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:48.822452 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:48.822479 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:48.823327 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:48.823811 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:48.823859 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.825381 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.825422 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:48.827168 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:48.827249 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:48.828342 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:48.828388 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:48.828668 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:48.829430 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:48.829497 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:48.833319 (+ 3822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1547,"thread_start_us":1789,"threads_started":1}
Aug 12 03:01:49 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:49 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:49.036929 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:49.017469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:59231)
0812 03:01:49.018710 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:49.018747 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:49.019016 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:49.019668 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:49.019697 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:49.020354 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:49.020749 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:49.020783 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.022211 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.022236 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:49.023366 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:49.023402 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.024476 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.024516 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:49.024881 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:49.031550 (+ 6669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:49.031638 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:49.035788 (+ 4150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":610,"threads_started":1}
W20260812 03:01:49.037017 25379 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:49.017571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:59231 (local address 127.10.118.126:41479)
0812 03:01:49.018061 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:49.018086 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:49.018872 (+ 786us) server_negotiation.cc:407] Connection header received
0812 03:01:49.019232 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:49.019253 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:49.019422 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:49.019584 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:49.020993 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.021976 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.023613 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.024189 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.024480 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:49.036115 (+ 11635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:59231: BlockingRecv error: recv got EOF from 127.10.118.66:59231 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 03:01:49 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:49 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:49.182929 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:49.169920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:43233)
0812 03:01:49.170934 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:49.170975 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:49.171186 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:49.171801 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:49.171820 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:49.172513 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:49.172852 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:49.172887 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.174467 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.174496 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:49.176018 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:49.176070 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.177269 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.177294 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:49.177553 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:49.178243 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:49.178303 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:49.181840 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":451,"threads_started":1}
W20260812 03:01:49.182999 25379 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:49.169998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:43233 (local address 127.10.118.126:41479)
0812 03:01:49.170632 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:49.170653 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:49.171089 (+ 436us) server_negotiation.cc:407] Connection header received
0812 03:01:49.171388 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:49.171407 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:49.171581 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:49.171714 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:49.173078 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.174250 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.176304 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.177040 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.177245 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:49.182185 (+ 4940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:43233: BlockingRecv error: recv got EOF from 127.10.118.65:43233 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 12 03:01:49 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:49 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:49.849646 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:49.836469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:35815)
0812 03:01:49.837693 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:49.837732 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:49.837984 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:49.838664 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:49.838684 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:49.839471 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:49.839787 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:49.839820 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.841137 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.841156 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:49.842344 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:49.842381 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.843371 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.843394 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:49.843684 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:49.844536 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:49.844620 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:49.848576 (+ 3956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":555,"threads_started":1}
W20260812 03:01:49.849783 25386 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:49.836534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:35815 (local address 127.10.118.126:41479)
0812 03:01:49.837716 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:49.837738 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:49.837831 (+ 93us) server_negotiation.cc:407] Connection header received
0812 03:01:49.838160 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:49.838177 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:49.838373 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:49.838517 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:49.840010 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.840955 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.842568 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:49.843150 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:49.843357 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:49.848864 (+ 5507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:35815: BlockingRecv error: recv got EOF from 127.10.118.67:35815 (error 108)
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":598,"threads_started":1}
Aug 12 03:01:50 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:50 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:50.052186 25386 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:50.038900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:52905 (local address 127.10.118.126:41479)
0812 03:01:50.039595 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:50.039641 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:50.040308 (+ 667us) server_negotiation.cc:407] Connection header received
0812 03:01:50.040633 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:50.040652 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:50.040836 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:50.040968 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:50.042173 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.043075 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.045004 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.045604 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.045869 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:50.050915 (+ 5046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:52905: BlockingRecv error: recv got EOF from 127.10.118.66:52905 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260812 03:01:50.052227 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:50.038776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:52905)
0812 03:01:50.040161 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:50.040199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:50.040462 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:50.041050 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:50.041069 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:50.041637 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:50.041956 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:50.041988 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.043450 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.043471 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:50.044715 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:50.044803 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.045859 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.045882 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:50.046123 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:50.046898 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:50.046960 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:50.050522 (+ 3562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":785,"threads_started":1}
Aug 12 03:01:50 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:50 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:50.199352 25386 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:50.185094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:47401 (local address 127.10.118.126:41479)
0812 03:01:50.185506 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:50.185524 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:50.186419 (+ 895us) server_negotiation.cc:407] Connection header received
0812 03:01:50.186724 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:50.186743 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:50.186915 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:50.187050 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:50.188616 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.189651 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.191403 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.192007 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.192289 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:50.198499 (+ 6210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:47401: BlockingRecv error: recv got EOF from 127.10.118.65:47401 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 03:01:50.199391 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:50.184982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:47401)
0812 03:01:50.186268 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:50.186305 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:50.186510 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:50.187240 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:50.187264 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:50.187956 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:50.188314 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:50.188351 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.189890 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.189910 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:50.191106 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:50.191144 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.192327 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.192354 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:50.192798 (+ 444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:50.193745 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:50.193838 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:50.198160 (+ 4322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":630,"threads_started":1}
Aug 12 03:01:50 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:50 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:50.866360 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:50.851336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:51013)
0812 03:01:50.852863 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:50.852918 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:50.853129 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:50.854058 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:50.854077 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:50.854652 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:50.854996 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:50.855029 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.856902 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.856926 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:50.858085 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:50.858122 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.859500 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.859525 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:50.859783 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:50.860597 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:50.860658 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:50.864712 (+ 4054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":2869,"threads_started":1}
W20260812 03:01:50.866605 25396 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:50.851511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:51013 (local address 127.10.118.126:41479)
0812 03:01:50.853372 (+ 1861us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:50.853393 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:50.853437 (+ 44us) server_negotiation.cc:407] Connection header received
0812 03:01:50.853560 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:50.853584 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:50.853796 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:50.853961 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:50.855254 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.856700 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.858342 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:50.859253 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:50.859522 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:50.865381 (+ 5859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:51013: BlockingRecv error: recv got EOF from 127.10.118.67:51013 (error 108)
Metrics: {"server-negotiator.queue_time_us":1458,"thread_start_us":1213,"threads_started":1}
Aug 12 03:01:51 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:51 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:51.067408 25396 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:51.054102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:48961 (local address 127.10.118.126:41479)
0812 03:01:51.054515 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:51.054533 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:51.055274 (+ 741us) server_negotiation.cc:407] Connection header received
0812 03:01:51.055559 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:51.055578 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:51.055754 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:51.055886 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:51.057128 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.058034 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.059804 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.060770 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.061084 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:51.066558 (+ 5474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:48961: BlockingRecv error: recv got EOF from 127.10.118.66:48961 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 03:01:51.067469 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:51.053888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:48961)
0812 03:01:51.055137 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:51.055176 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:51.055393 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:51.055974 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:51.055992 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:51.056582 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:51.056902 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:51.056935 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.058219 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.058238 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:51.059510 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:51.059560 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.061088 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.061122 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:51.061454 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:51.062302 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:51.062363 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:51.066186 (+ 3823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":600,"threads_started":1}
Aug 12 03:01:51 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:51 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:51.216814 25396 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:51.201326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:35089 (local address 127.10.118.126:41479)
0812 03:01:51.201793 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:51.201808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:51.202778 (+ 970us) server_negotiation.cc:407] Connection header received
0812 03:01:51.203203 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:51.203232 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:51.203433 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:51.203607 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:51.204996 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.205904 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.207570 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.208135 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.208490 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:51.215495 (+ 7005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:35089: BlockingRecv error: recv got EOF from 127.10.118.65:35089 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 03:01:51.216841 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:51.201233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:35089)
0812 03:01:51.202571 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:51.202637 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:51.202907 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:51.203701 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:51.203737 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:51.204419 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:51.204783 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:51.204816 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.206090 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.206110 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:51.207356 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:51.207393 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.208459 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.208482 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:51.208792 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:51.209652 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:51.209788 (+ 136us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:51.215047 (+ 5259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":683,"threads_started":1}
Aug 12 03:01:51 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:51 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:51.881728 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:51.867901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:32893)
0812 03:01:51.869311 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:51.869364 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:51.869618 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:51.870372 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:51.870401 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:51.871047 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:51.871373 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:51.871405 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.872852 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.872873 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:51.874417 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:51.874458 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.875659 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.875686 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:51.876024 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:51.877046 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:51.877109 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:51.880645 (+ 3536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":562,"threads_started":1}
W20260812 03:01:51.881871 25405 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:51.868075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:32893 (local address 127.10.118.126:41479)
0812 03:01:51.869403 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:51.869430 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:51.869480 (+ 50us) server_negotiation.cc:407] Connection header received
0812 03:01:51.869866 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:51.869885 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:51.870085 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:51.870278 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:51.871635 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.872628 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.874720 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:51.875404 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:51.875693 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:51.880976 (+ 5283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:32893: BlockingRecv error: recv got EOF from 127.10.118.67:32893 (error 108)
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":555,"threads_started":1}
Aug 12 03:01:52 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:52 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:52.083019 25405 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:52.069886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:34569 (local address 127.10.118.126:41479)
0812 03:01:52.070359 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:52.070388 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:52.071202 (+ 814us) server_negotiation.cc:407] Connection header received
0812 03:01:52.071468 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:52.071487 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:52.071657 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:52.071801 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:52.073180 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.074067 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.075765 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.076340 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.076604 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:52.082002 (+ 5398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:34569: BlockingRecv error: recv got EOF from 127.10.118.66:34569 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 03:01:52.083431 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:52.069785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:34569)
0812 03:01:52.071044 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:52.071092 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:52.071296 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:52.071900 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:52.071920 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:52.072627 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:52.072946 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:52.072979 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.074286 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.074308 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:52.075522 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:52.075560 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.076641 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.076665 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:52.076923 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:52.077763 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:52.077826 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:52.081674 (+ 3848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":591,"threads_started":1}
Aug 12 03:01:52 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:52 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:52.232345 25405 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:52.218726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:46717 (local address 127.10.118.126:41479)
0812 03:01:52.219120 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:52.219138 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:52.220076 (+ 938us) server_negotiation.cc:407] Connection header received
0812 03:01:52.220466 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:52.220484 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:52.220659 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:52.220798 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:52.222071 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.222970 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.224721 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.225469 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.225765 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:52.231070 (+ 5305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:46717: BlockingRecv error: recv got EOF from 127.10.118.65:46717 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 03:01:52.232359 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:52.218648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:46717)
0812 03:01:52.219936 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:52.219974 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:52.220191 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:52.220895 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:52.220915 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:52.221525 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:52.221855 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:52.221890 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.223186 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.223220 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:52.224476 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:52.224526 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.225805 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.225837 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:52.226198 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:52.227033 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:52.227093 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:52.230651 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":663,"threads_started":1}
Aug 12 03:01:52 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:52 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:52.897538 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:52.883346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:58625)
0812 03:01:52.884712 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:52.884755 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:52.884952 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:52.885732 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:52.885751 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:52.886340 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:52.886666 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:52.886699 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.888856 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.888876 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:52.890358 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:52.890397 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.891463 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.891487 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:52.891752 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:52.892590 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:52.892684 (+ 94us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:52.896455 (+ 3771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":515,"threads_started":1}
W20260812 03:01:52.898150 25413 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:52.883717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:58625 (local address 127.10.118.126:41479)
0812 03:01:52.885011 (+ 1294us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:52.885031 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:52.885079 (+ 48us) server_negotiation.cc:407] Connection header received
0812 03:01:52.885248 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:52.885264 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:52.885468 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:52.885618 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:52.886916 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.888596 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.890582 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:52.891210 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:52.891419 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:52.896834 (+ 5415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:58625: BlockingRecv error: recv got EOF from 127.10.118.67:58625 (error 108)
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":599,"threads_started":1}
Aug 12 03:01:53 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:53 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:53.098040 25418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:53.085153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:56329)
0812 03:01:53.086132 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:53.086170 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:53.086373 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:53.086947 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:53.086966 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:53.087528 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:53.087851 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:53.087884 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.089180 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.089200 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:53.090340 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:53.090376 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.091363 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.091387 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:53.091632 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:53.092394 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:53.092456 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:53.096283 (+ 3827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":403,"threads_started":1}
W20260812 03:01:53.098028 25413 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:53.085283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:56329 (local address 127.10.118.126:41479)
0812 03:01:53.085853 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:53.085878 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:53.086287 (+ 409us) server_negotiation.cc:407] Connection header received
0812 03:01:53.086544 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:53.086562 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:53.086729 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:53.086860 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:53.088071 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.088985 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.090568 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.091138 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.091338 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:53.096740 (+ 5402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:56329: BlockingRecv error: recv got EOF from 127.10.118.66:56329 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 12 03:01:53 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:53 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:53.248101 25419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:53.234069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:60851)
0812 03:01:53.235063 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:53.235100 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:53.235310 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:53.235955 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:53.235974 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:53.236574 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:53.237081 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:53.237139 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.238991 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.239021 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:53.240631 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:53.240672 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.241681 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.241704 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:53.241958 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:53.242649 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:53.242709 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:53.246413 (+ 3704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":435,"threads_started":1}
W20260812 03:01:53.248220 25413 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:53.234166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:60851 (local address 127.10.118.126:41479)
0812 03:01:53.234755 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:53.234778 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:53.235223 (+ 445us) server_negotiation.cc:407] Connection header received
0812 03:01:53.235521 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:53.235540 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:53.235737 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:53.235867 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:53.237377 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.238737 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.240857 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.241460 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.241659 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:53.246823 (+ 5164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:60851: BlockingRecv error: recv got EOF from 127.10.118.65:60851 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Aug 12 03:01:53 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:53 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.67@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:53.914799 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:53.899508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.67:40455)
0812 03:01:53.900976 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:53.901026 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:53.901274 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:53.902037 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:53.902062 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:53.903032 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:53.903421 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:53.903457 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.905225 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.905260 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:53.906504 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:53.906542 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.907794 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.907826 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:53.908253 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:53.909066 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:53.909128 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:53.913245 (+ 4117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":568,"threads_started":1}
W20260812 03:01:53.915405 25421 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:53.899948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:40455 (local address 127.10.118.126:41479)
0812 03:01:53.901197 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:53.901228 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:53.901274 (+ 46us) server_negotiation.cc:407] Connection header received
0812 03:01:53.901515 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:53.901541 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:53.901778 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:53.901957 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:53.903667 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.904956 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.906746 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:53.907508 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:53.907788 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:53.914046 (+ 6258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.67:40455: BlockingRecv error: recv got EOF from 127.10.118.67:40455 (error 108)
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":555,"threads_started":1}
Aug 12 03:01:54 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:54 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.66@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
I20260812 03:01:54.118213 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:54.099651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.66:48165)
0812 03:01:54.100969 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:54.101008 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:54.101217 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:54.101828 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:54.101848 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:54.102441 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:54.102784 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:54.102817 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.104258 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.104282 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:54.105513 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:54.105551 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.106615 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.106642 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:54.106992 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:54.113418 (+ 6426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:54.113480 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:54.117162 (+ 3682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":663,"threads_started":1}
W20260812 03:01:54.118439 25421 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:54.099760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:48165 (local address 127.10.118.126:41479)
0812 03:01:54.100190 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:54.100213 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:54.101123 (+ 910us) server_negotiation.cc:407] Connection header received
0812 03:01:54.101389 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:54.101414 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:54.101584 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:54.101724 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:54.103004 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.104039 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.105749 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.106325 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.106542 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:54.117555 (+ 11013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.66:48165: BlockingRecv error: recv got EOF from 127.10.118.66:48165 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 12 03:01:54 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
Aug 12 03:01:54 dist-test-slave-zpvx krb5kdc[23374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.118.65@KRBTEST.COM for oryx/127.10.118.126@KRBTEST.COM, Server not found in Kerberos database
W20260812 03:01:54.264364 25421 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:01:54.250053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:53947 (local address 127.10.118.126:41479)
0812 03:01:54.250482 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:54.250497 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:54.251244 (+ 747us) server_negotiation.cc:407] Connection header received
0812 03:01:54.251605 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:54.251626 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:54.251800 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:54.251936 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:54.253691 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.254793 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.256959 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.257546 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.257751 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:54.263083 (+ 5332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.118.65:53947: BlockingRecv error: recv got EOF from 127.10.118.65:53947 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 03:01:54.264415 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:54.249854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41479 (local address 127.10.118.65:53947)
0812 03:01:54.251091 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:54.251130 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:54.251378 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:54.252084 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:54.252113 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:54.252957 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:54.253391 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:54.253440 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.255026 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.255055 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:54.256718 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:54.256757 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:54.257772 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:54.257795 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:54.258033 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:54.258766 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:01:54.258825 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:54.262688 (+ 3863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.118.126:41479: Server oryx/127.10.118.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":620,"threads_started":1}
I20260812 03:01:54.640925 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 24491
I20260812 03:01:54.679843 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 24789
I20260812 03:01:54.714110 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 25036
I20260812 03:01:54.745369 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 24343
2026-08-12T03:01:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (32364 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T03:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:01:54Z Disabled control of system clock
I20260812 03:01:54.850796 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:42239
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 03:01:55.159194 25433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:55.159765 25433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:55.160461 25433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:55.191987 25433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:01:55.192286 25433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:55.192526 25433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:55.192744 25433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:01:55.192931 25433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:01:55.228222 25433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:42239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:55.229501 25433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:55.231172 25433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:55.248690 25442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:55.248703 25445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:55.250756 25433 server_base.cc:1034] running on GCE node
W20260812 03:01:55.248873 25441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:56.499166 25433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:56.502516 25433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:56.503981 25433 hybrid_clock.cc:648] HybridClock initialized: now 1786503716503934 us; error 60 us; skew 500 ppm
I20260812 03:01:56.504910 25433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:01:56.511833 25433 webserver.cc:460] Webserver started at http://127.10.118.126:44905/ using document root <none> and password file <none>
I20260812 03:01:56.512856 25433 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:56.513095 25433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:56.513582 25433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:56.519127 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "8760da6dea704621b1f0c138d0062b26"
format_stamp: "Formatted at 2026-08-12 03:01:56 on dist-test-slave-zpvx"
server_key: "afc880331606c6dc55592cfa51a4f28e"
server_key_iv: "d3e115c33ed8b31989f13cc11860843e"
server_key_version: "encryptionkey@0"
I20260812 03:01:56.520320 25433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "8760da6dea704621b1f0c138d0062b26"
format_stamp: "Formatted at 2026-08-12 03:01:56 on dist-test-slave-zpvx"
server_key: "afc880331606c6dc55592cfa51a4f28e"
server_key_iv: "d3e115c33ed8b31989f13cc11860843e"
server_key_version: "encryptionkey@0"
I20260812 03:01:56.527647 25433 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 03:01:56.533514 25456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:56.534590 25433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 03:01:56.534947 25433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "8760da6dea704621b1f0c138d0062b26"
format_stamp: "Formatted at 2026-08-12 03:01:56 on dist-test-slave-zpvx"
server_key: "afc880331606c6dc55592cfa51a4f28e"
server_key_iv: "d3e115c33ed8b31989f13cc11860843e"
server_key_version: "encryptionkey@0"
I20260812 03:01:56.535324 25433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:56.573987 25433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:56.575665 25433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:56.576138 25433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:56.647152 25433 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:42239
I20260812 03:01:56.647244 25549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:42239 every 8 connection(s)
I20260812 03:01:56.650058 25433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:01:56.655253 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:01:56.657286 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 25433
I20260812 03:01:56.657836 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:01:56.659616 10713 external_mini_cluster.cc:1428] Setting key 85e2aa193c2cecf67f7306d07b8ed8a4
I20260812 03:01:56.681360 25551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26: Bootstrap starting.
I20260812 03:01:56.682973 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:56.662666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.10.118.126:42239)
0812 03:01:56.665813 (+ 3147us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:56.665839 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:56.665894 (+ 55us) server_negotiation.cc:407] Connection header received
0812 03:01:56.669942 (+ 4048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:56.669984 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:56.670367 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:56.670697 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:56.672236 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:56.673906 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:56.676091 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:56.677017 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:56.677371 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:56.679163 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:56.679196 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:56.679224 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:01:56.679967 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:56.680072 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:01:56.680114 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:56.680284 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:56.680689 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 03:01:56.681063 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2596,"thread_start_us":2024,"threads_started":1}
I20260812 03:01:56.689303 25551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26: Neither blocks nor log segments found. Creating new log.
I20260812 03:01:56.692270 25551 log.cc:824] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26: Log is configured to *not* fsync() on all Append() calls
I20260812 03:01:56.697803 25551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26: No bootstrap required, opened a new log
I20260812 03:01:56.725448 25551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760da6dea704621b1f0c138d0062b26" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42239 } }
I20260812 03:01:56.726177 25551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:01:56.726461 25551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8760da6dea704621b1f0c138d0062b26, State: Initialized, Role: FOLLOWER
I20260812 03:01:56.727232 25551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760da6dea704621b1f0c138d0062b26" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42239 } }
I20260812 03:01:56.727783 25551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:01:56.728061 25551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:01:56.728470 25551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:01:56.733165 25551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760da6dea704621b1f0c138d0062b26" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42239 } }
I20260812 03:01:56.733919 25551 leader_election.cc:302] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8760da6dea704621b1f0c138d0062b26; no voters:
I20260812 03:01:56.735635 25551 leader_election.cc:288] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:01:56.736040 25565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:01:56.738240 25565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [term 1 LEADER]: Becoming Leader. State: Replica: 8760da6dea704621b1f0c138d0062b26, State: Running, Role: LEADER
I20260812 03:01:56.739084 25565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760da6dea704621b1f0c138d0062b26" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42239 } }
I20260812 03:01:56.740103 25551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:01:56.751348 25567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8760da6dea704621b1f0c138d0062b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760da6dea704621b1f0c138d0062b26" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42239 } } }
I20260812 03:01:56.751231 25568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8760da6dea704621b1f0c138d0062b26. Latest consensus state: current_term: 1 leader_uuid: "8760da6dea704621b1f0c138d0062b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8760da6dea704621b1f0c138d0062b26" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42239 } } }
I20260812 03:01:56.752077 25568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:56.752077 25567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26 [sys.catalog]: This master's current role is: LEADER
I20260812 03:01:56.759488 25579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:01:56.770702 25579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:01:56.787787 25579 catalog_manager.cc:1347] Generated new cluster ID: 4dc4c9582eef4fe7ae7171f799d69f36
I20260812 03:01:56.788086 25579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:01:56.812546 25579 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:01:56.815101 25579 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:01:56.829030 25579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8760da6dea704621b1f0c138d0062b26: Generated new TSK 0
I20260812 03:01:56.830006 25579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:01:56.853649 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42239
--builtin_ntp_servers=127.10.118.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 03:01:57.167052 25599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:57.167603 25599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:57.168144 25599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:57.200003 25599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:57.200294 25599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:57.201201 25599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:01:57.235352 25599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:57.236672 25599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:57.238310 25599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:57.260457 25610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:57.262799 25599 server_base.cc:1034] running on GCE node
W20260812 03:01:57.265960 25618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:57.261480 25609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:58.456738 25599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:01:58.459597 25599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:01:58.461048 25599 hybrid_clock.cc:648] HybridClock initialized: now 1786503718460975 us; error 77 us; skew 500 ppm
I20260812 03:01:58.462109 25599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:01:58.469940 25599 webserver.cc:460] Webserver started at http://127.10.118.65:38993/ using document root <none> and password file <none>
I20260812 03:01:58.471186 25599 fs_manager.cc:359] Metadata directory not provided
I20260812 03:01:58.471450 25599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:01:58.472031 25599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:01:58.479961 25599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "845a07378cbb4cb4aea1ad4b6490d1f4"
format_stamp: "Formatted at 2026-08-12 03:01:58 on dist-test-slave-zpvx"
server_key: "2bc7074dadf8fababdb477081851df5f"
server_key_iv: "94164e368b67a5ff235fc2592c8d5332"
server_key_version: "encryptionkey@0"
I20260812 03:01:58.481600 25599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "845a07378cbb4cb4aea1ad4b6490d1f4"
format_stamp: "Formatted at 2026-08-12 03:01:58 on dist-test-slave-zpvx"
server_key: "2bc7074dadf8fababdb477081851df5f"
server_key_iv: "94164e368b67a5ff235fc2592c8d5332"
server_key_version: "encryptionkey@0"
I20260812 03:01:58.489856 25599 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:01:58.495631 25633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:58.496784 25599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:01:58.497112 25599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "845a07378cbb4cb4aea1ad4b6490d1f4"
format_stamp: "Formatted at 2026-08-12 03:01:58 on dist-test-slave-zpvx"
server_key: "2bc7074dadf8fababdb477081851df5f"
server_key_iv: "94164e368b67a5ff235fc2592c8d5332"
server_key_version: "encryptionkey@0"
I20260812 03:01:58.497499 25599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:01:58.531128 25599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:01:58.532770 25599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:01:58.533258 25599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:01:58.536484 25599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:01:58.541599 25599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:01:58.541833 25599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:58.542119 25599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:01:58.542303 25599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:01:58.702775 25599 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:44241
I20260812 03:01:58.702876 25862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:44241 every 8 connection(s)
I20260812 03:01:58.705446 25599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:01:58.713160 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 25599
I20260812 03:01:58.713621 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:01:58.714882 10713 external_mini_cluster.cc:1428] Setting key 01ed2d6787d2d090979e5d22327bf575
I20260812 03:01:58.720175 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42239
--builtin_ntp_servers=127.10.118.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 03:01:58.729853 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:58.709923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:47877 (local address 127.10.118.126:42239)
0812 03:01:58.711117 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0812 03:01:58.711136 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:01:58.713884 (+ 2748us) server_negotiation.cc:407] Connection header received
0812 03:01:58.718982 (+ 5098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:01:58.719015 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:01:58.719198 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:01:58.719377 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:01:58.721163 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:58.722071 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:58.724176 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:01:58.724831 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:01:58.725115 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:58.727009 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:01:58.727031 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:01:58.727048 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:01:58.727092 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:01:58.727154 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:01:58.727192 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:01:58.727367 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 03:01:58.727697 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 03:01:58.727911 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":560,"threads_started":1}
I20260812 03:01:58.730737 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:01:58.710221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42239 (local address 127.10.118.65:47877)
0812 03:01:58.713620 (+ 3399us) negotiation.cc:107] Waiting for socket to connect
0812 03:01:58.713687 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 03:01:58.718618 (+ 4931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:01:58.719665 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:01:58.719702 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:01:58.719870 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:01:58.720892 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:58.720933 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:58.722333 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:58.722353 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:58.723926 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:01:58.723962 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:01:58.725161 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:01:58.725192 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:01:58.725623 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:01:58.726499 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:01:58.726512 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:01:58.726573 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:01:58.726691 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:01:58.727383 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:01:58.727436 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:01:58.727460 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 03:01:58.727653 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 03:01:58.728122 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2666,"thread_start_us":1768,"threads_started":1}
I20260812 03:01:58.732708 25864 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42239
I20260812 03:01:58.733120 25864 heartbeater.cc:464] Registering TS with master...
I20260812 03:01:58.734031 25864 heartbeater.cc:511] Master 127.10.118.126:42239 requested a full tablet report, sending...
I20260812 03:01:58.736645 25488 ts_manager.cc:194] Registered new tserver with Master: 845a07378cbb4cb4aea1ad4b6490d1f4 (127.10.118.65:44241)
I20260812 03:01:58.738679 25488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.65:47877
W20260812 03:01:59.032522 25871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:01:59.033143 25871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:01:59.033687 25871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:01:59.064597 25871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:01:59.064869 25871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:01:59.065706 25871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:01:59.101727 25871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:01:59.102944 25871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:01:59.104714 25871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:01:59.124784 25878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:59.742797 25864 heartbeater.cc:503] Master 127.10.118.126:42239 was elected leader, sending a full tablet report...
W20260812 03:01:59.124809 25883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:01:59.124989 25880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:01:59.126307 25871 server_base.cc:1034] running on GCE node
I20260812 03:02:00.288961 25871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:00.291301 25871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:00.292673 25871 hybrid_clock.cc:648] HybridClock initialized: now 1786503720292628 us; error 57 us; skew 500 ppm
I20260812 03:02:00.293536 25871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:02:00.304414 25871 webserver.cc:460] Webserver started at http://127.10.118.66:38411/ using document root <none> and password file <none>
I20260812 03:02:00.305480 25871 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:00.305749 25871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:00.306259 25871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:00.311734 25871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "4e449ee4aa45416bb28616e9d1f78479"
format_stamp: "Formatted at 2026-08-12 03:02:00 on dist-test-slave-zpvx"
server_key: "724cce4a2b17a660be9696ba13a4604a"
server_key_iv: "f677e3217cb8407d0664ed13674be542"
server_key_version: "encryptionkey@0"
I20260812 03:02:00.313062 25871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "4e449ee4aa45416bb28616e9d1f78479"
format_stamp: "Formatted at 2026-08-12 03:02:00 on dist-test-slave-zpvx"
server_key: "724cce4a2b17a660be9696ba13a4604a"
server_key_iv: "f677e3217cb8407d0664ed13674be542"
server_key_version: "encryptionkey@0"
I20260812 03:02:00.321382 25871 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260812 03:02:00.327276 25893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:00.328322 25871 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 03:02:00.328742 25871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "4e449ee4aa45416bb28616e9d1f78479"
format_stamp: "Formatted at 2026-08-12 03:02:00 on dist-test-slave-zpvx"
server_key: "724cce4a2b17a660be9696ba13a4604a"
server_key_iv: "f677e3217cb8407d0664ed13674be542"
server_key_version: "encryptionkey@0"
I20260812 03:02:00.329209 25871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:00.374030 25871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:00.375751 25871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:00.376276 25871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:00.379123 25871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:00.384325 25871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:00.384622 25871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 03:02:00.384938 25871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:00.385150 25871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:00.538002 25871 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:45231
I20260812 03:02:00.538125 26089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:45231 every 8 connection(s)
I20260812 03:02:00.540820 25871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:02:00.550079 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 25871
I20260812 03:02:00.550558 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:02:00.551601 10713 external_mini_cluster.cc:1428] Setting key 5866e460013d8c4a94bcbc90398e4a60
I20260812 03:02:00.557765 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42239
--builtin_ntp_servers=127.10.118.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 03:02:00.566489 26098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:00.545188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:33495 (local address 127.10.118.126:42239)
0812 03:02:00.546176 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:00.546196 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:00.548362 (+ 2166us) server_negotiation.cc:407] Connection header received
0812 03:02:00.553632 (+ 5270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:00.553660 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:00.553875 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:00.554070 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:00.556239 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:00.557228 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:00.560613 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:00.561355 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:00.561678 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:00.564160 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:00.564183 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:00.564201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:02:00.564251 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:00.564324 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:02:00.564416 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:00.564563 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:00.565008 (+ 445us) server_negotiation.cc:299] Negotiation successful
0812 03:02:00.565198 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":416,"threads_started":1}
I20260812 03:02:00.568032 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:00.545544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42239 (local address 127.10.118.66:33495)
0812 03:02:00.548067 (+ 2523us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:00.548206 (+ 139us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:00.553281 (+ 5075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:00.554406 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:00.554440 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:00.554631 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:00.555939 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:00.556000 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:00.557502 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:00.557533 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:00.560271 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:00.560338 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:00.561771 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:00.561809 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:00.562408 (+ 599us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:00.563495 (+ 1087us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:02:00.563523 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:00.563613 (+ 90us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:02:00.563797 (+ 184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:00.564614 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:00.564684 (+ 70us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:00.564723 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 03:02:00.564970 (+ 247us) client_negotiation.cc:240] Negotiation successful
0812 03:02:00.565491 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1855,"thread_start_us":1146,"threads_started":1}
I20260812 03:02:00.570184 26092 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42239
I20260812 03:02:00.570730 26092 heartbeater.cc:464] Registering TS with master...
I20260812 03:02:00.572183 26092 heartbeater.cc:511] Master 127.10.118.126:42239 requested a full tablet report, sending...
I20260812 03:02:00.574801 25488 ts_manager.cc:194] Registered new tserver with Master: 4e449ee4aa45416bb28616e9d1f78479 (127.10.118.66:45231)
I20260812 03:02:00.576256 25488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.66:33495
W20260812 03:02:00.892902 26103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:00.893466 26103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:00.894032 26103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:00.927047 26103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:00.927356 26103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:00.928213 26103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:02:00.962859 26103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:00.964097 26103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:00.965758 26103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:00.983680 26115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:01.581116 26092 heartbeater.cc:503] Master 127.10.118.126:42239 was elected leader, sending a full tablet report...
W20260812 03:02:00.983801 26112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:00.984207 26113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:02.161382 26114 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:02:02.161545 26103 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:02:02.165411 26103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:02.168232 26103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:02.169687 26103 hybrid_clock.cc:648] HybridClock initialized: now 1786503722169657 us; error 49 us; skew 500 ppm
I20260812 03:02:02.170481 26103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:02:02.176909 26103 webserver.cc:460] Webserver started at http://127.10.118.67:37977/ using document root <none> and password file <none>
I20260812 03:02:02.177839 26103 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:02.178026 26103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:02.178447 26103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:02.183823 26103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "3b59ac2da1064949855754138ca8ab6b"
format_stamp: "Formatted at 2026-08-12 03:02:02 on dist-test-slave-zpvx"
server_key: "7c2d83970d291a18f0c90e0a3436eca9"
server_key_iv: "20961ec4cffb8df0c0699cdaeae5bef5"
server_key_version: "encryptionkey@0"
I20260812 03:02:02.185055 26103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "3b59ac2da1064949855754138ca8ab6b"
format_stamp: "Formatted at 2026-08-12 03:02:02 on dist-test-slave-zpvx"
server_key: "7c2d83970d291a18f0c90e0a3436eca9"
server_key_iv: "20961ec4cffb8df0c0699cdaeae5bef5"
server_key_version: "encryptionkey@0"
I20260812 03:02:02.192545 26103 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:02:02.198299 26124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:02.199393 26103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:02:02.199740 26103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "3b59ac2da1064949855754138ca8ab6b"
format_stamp: "Formatted at 2026-08-12 03:02:02 on dist-test-slave-zpvx"
server_key: "7c2d83970d291a18f0c90e0a3436eca9"
server_key_iv: "20961ec4cffb8df0c0699cdaeae5bef5"
server_key_version: "encryptionkey@0"
I20260812 03:02:02.200141 26103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:02.234004 26103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:02.235536 26103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:02.236006 26103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:02.238608 26103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:02.243678 26103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:02.243920 26103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:02.244206 26103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:02.244395 26103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:02.393002 26103 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:35217
I20260812 03:02:02.393106 26308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:35217 every 8 connection(s)
I20260812 03:02:02.395835 26103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:02:02.404249 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 26103
I20260812 03:02:02.404788 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:02:02.405833 10713 external_mini_cluster.cc:1428] Setting key 5607a9bd27033032dae324201e1cc683
I20260812 03:02:02.421092 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:02.400273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:40489 (local address 127.10.118.126:42239)
0812 03:02:02.401511 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:02.401534 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:02.404262 (+ 2728us) server_negotiation.cc:407] Connection header received
0812 03:02:02.409827 (+ 5565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:02.409861 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:02.410047 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:02.410228 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:02.411978 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:02.413033 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:02.415141 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:02.415768 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:02.416056 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:02.418134 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:02.418155 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:02.418171 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:02:02.418217 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:02.418305 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:02:02.418339 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:02.418493 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:02.418859 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 03:02:02.419073 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":562,"threads_started":1}
I20260812 03:02:02.421784 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:02.400686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42239 (local address 127.10.118.67:40489)
0812 03:02:02.404012 (+ 3326us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:02.404078 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:02.409475 (+ 5397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:02.410500 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:02.410536 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:02.410695 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:02.411688 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:02.411727 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:02.413288 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:02.413309 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:02.414874 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:02.414916 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:02.416125 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:02.416158 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:02.416675 (+ 517us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:02.417606 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:02:02.417619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:02.417684 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:02:02.417823 (+ 139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:02.418507 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:02.418559 (+ 52us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:02.418592 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 03:02:02.418810 (+ 218us) client_negotiation.cc:240] Negotiation successful
0812 03:02:02.419235 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2605,"thread_start_us":1736,"threads_started":1}
I20260812 03:02:02.423653 26311 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42239
I20260812 03:02:02.424072 26311 heartbeater.cc:464] Registering TS with master...
I20260812 03:02:02.425199 26311 heartbeater.cc:511] Master 127.10.118.126:42239 requested a full tablet report, sending...
I20260812 03:02:02.427417 25488 ts_manager.cc:194] Registered new tserver with Master: 3b59ac2da1064949855754138ca8ab6b (127.10.118.67:35217)
I20260812 03:02:02.429100 25488 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.67:40489
I20260812 03:02:02.439874 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 03:02:02.457261 26315 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:02:02.453051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.10.118.126:42239)
0812 03:02:02.453470 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:02.453489 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:02.454576 (+ 1087us) server_negotiation.cc:407] Connection header received
0812 03:02:02.454996 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:02.455016 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:02.455189 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:02.455396 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:02.456176 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46462: BlockingRecv error: recv got EOF from 127.0.0.1:46462 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 03:02:02.458005 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 25599
I20260812 03:02:02.506445 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 25871
I20260812 03:02:02.547278 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 26103
I20260812 03:02:02.583314 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 25433
2026-08-12T03:02:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (7829 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T03:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:02:02Z Disabled control of system clock
I20260812 03:02:02.682504 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:40241
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 03:02:03.000058 26331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:03.000777 26331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:03.001367 26331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:03.032749 26331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:02:03.033123 26331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:03.033342 26331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:03.033618 26331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:02:03.033879 26331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:02:03.069388 26331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:40241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:03.070631 26331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:03.072268 26331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:03.088905 26345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:03.088980 26349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:03.089520 26342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:03.090590 26331 server_base.cc:1034] running on GCE node
I20260812 03:02:04.272279 26331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:04.274964 26331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:04.276330 26331 hybrid_clock.cc:648] HybridClock initialized: now 1786503724276299 us; error 52 us; skew 500 ppm
I20260812 03:02:04.277235 26331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:02:04.283650 26331 webserver.cc:460] Webserver started at http://127.10.118.126:45499/ using document root <none> and password file <none>
I20260812 03:02:04.284644 26331 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:04.284833 26331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:04.285413 26331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:04.293280 26331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "8a8e7be052624e409b7b922b1a0e445c"
format_stamp: "Formatted at 2026-08-12 03:02:04 on dist-test-slave-zpvx"
server_key: "63f7b5d1dc9fdc2ec6b802dfd9108377"
server_key_iv: "e353a1f64a08d279ef0610d716ca893e"
server_key_version: "encryptionkey@0"
I20260812 03:02:04.294394 26331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "8a8e7be052624e409b7b922b1a0e445c"
format_stamp: "Formatted at 2026-08-12 03:02:04 on dist-test-slave-zpvx"
server_key: "63f7b5d1dc9fdc2ec6b802dfd9108377"
server_key_iv: "e353a1f64a08d279ef0610d716ca893e"
server_key_version: "encryptionkey@0"
I20260812 03:02:04.302131 26331 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260812 03:02:04.307898 26361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:04.309065 26331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:02:04.309367 26331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "8a8e7be052624e409b7b922b1a0e445c"
format_stamp: "Formatted at 2026-08-12 03:02:04 on dist-test-slave-zpvx"
server_key: "63f7b5d1dc9fdc2ec6b802dfd9108377"
server_key_iv: "e353a1f64a08d279ef0610d716ca893e"
server_key_version: "encryptionkey@0"
I20260812 03:02:04.309701 26331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:04.336095 26331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:04.337597 26331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:04.338008 26331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:04.408125 26331 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:40241
I20260812 03:02:04.408241 26456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:40241 every 8 connection(s)
I20260812 03:02:04.410925 26331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:02:04.412453 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 26331
I20260812 03:02:04.413384 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:02:04.415297 10713 external_mini_cluster.cc:1428] Setting key 49dd9ffbf6b5f604ec9228f5f33aa95d
I20260812 03:02:04.417457 26458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:02:04.434031 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:04.418079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.10.118.126:40241)
0812 03:02:04.420873 (+ 2794us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:04.420910 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:04.420985 (+ 75us) server_negotiation.cc:407] Connection header received
0812 03:02:04.428155 (+ 7170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:04.428218 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:04.428692 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:04.429198 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:04.430772 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:04.430806 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:04.430847 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:02:04.430907 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:04.431027 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:02:04.431091 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:04.431392 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:04.431550 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 03:02:04.431817 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2065,"thread_start_us":1682,"threads_started":1}
I20260812 03:02:04.445204 26458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c: Bootstrap starting.
I20260812 03:02:04.450582 26458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c: Neither blocks nor log segments found. Creating new log.
I20260812 03:02:04.452157 26458 log.cc:824] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c: Log is configured to *not* fsync() on all Append() calls
I20260812 03:02:04.456641 26458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c: No bootstrap required, opened a new log
I20260812 03:02:04.475622 26458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e7be052624e409b7b922b1a0e445c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:02:04.476354 26458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:02:04.476657 26458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8e7be052624e409b7b922b1a0e445c, State: Initialized, Role: FOLLOWER
I20260812 03:02:04.477591 26458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e7be052624e409b7b922b1a0e445c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:02:04.478170 26458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:02:04.478410 26458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:02:04.478658 26458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:02:04.482787 26458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e7be052624e409b7b922b1a0e445c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:02:04.483430 26458 leader_election.cc:302] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a8e7be052624e409b7b922b1a0e445c; no voters:
I20260812 03:02:04.485208 26458 leader_election.cc:288] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:02:04.486037 26467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:02:04.490268 26467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [term 1 LEADER]: Becoming Leader. State: Replica: 8a8e7be052624e409b7b922b1a0e445c, State: Running, Role: LEADER
I20260812 03:02:04.490494 26458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:02:04.491223 26467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e7be052624e409b7b922b1a0e445c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } }
I20260812 03:02:04.499682 26468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a8e7be052624e409b7b922b1a0e445c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e7be052624e409b7b922b1a0e445c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } } }
I20260812 03:02:04.500633 26468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [sys.catalog]: This master's current role is: LEADER
I20260812 03:02:04.499881 26470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a8e7be052624e409b7b922b1a0e445c. Latest consensus state: current_term: 1 leader_uuid: "8a8e7be052624e409b7b922b1a0e445c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8e7be052624e409b7b922b1a0e445c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 40241 } } }
I20260812 03:02:04.501104 26470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c [sys.catalog]: This master's current role is: LEADER
I20260812 03:02:04.508311 26486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:02:04.518481 26486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:02:04.535631 26486 catalog_manager.cc:1347] Generated new cluster ID: dd3675f498864f36bb81e9a4da4d617f
I20260812 03:02:04.535892 26486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:02:04.564268 26486 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:02:04.564939 26486 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:02:04.579510 26486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a8e7be052624e409b7b922b1a0e445c: Generated new TSK 0
I20260812 03:02:04.580458 26486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:02:04.595969 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40241
--builtin_ntp_servers=127.10.118.84:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 03:02:04.921906 26516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:04.922429 26516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:04.923026 26516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:04.954313 26516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:04.954577 26516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:04.955504 26516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:02:04.990172 26516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:04.991426 26516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:04.993162 26516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:05.014783 26529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:05.015384 26530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:05.016959 26532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:05.016992 26516 server_base.cc:1034] running on GCE node
I20260812 03:02:06.221969 26516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:06.224323 26516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:06.225790 26516 hybrid_clock.cc:648] HybridClock initialized: now 1786503726225760 us; error 48 us; skew 500 ppm
I20260812 03:02:06.226727 26516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:02:06.238286 26516 webserver.cc:460] Webserver started at http://127.10.118.65:33299/ using document root <none> and password file <none>
I20260812 03:02:06.239274 26516 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:06.239513 26516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:06.239995 26516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:06.245488 26516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "e520e20c0be1483aa73a80b6b66fef79"
format_stamp: "Formatted at 2026-08-12 03:02:06 on dist-test-slave-zpvx"
server_key: "e2c56666eabd930f564877a213312aa6"
server_key_iv: "e644bddeb3b0fc4b48e1af7e6013f5d5"
server_key_version: "encryptionkey@0"
I20260812 03:02:06.246663 26516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "e520e20c0be1483aa73a80b6b66fef79"
format_stamp: "Formatted at 2026-08-12 03:02:06 on dist-test-slave-zpvx"
server_key: "e2c56666eabd930f564877a213312aa6"
server_key_iv: "e644bddeb3b0fc4b48e1af7e6013f5d5"
server_key_version: "encryptionkey@0"
I20260812 03:02:06.255366 26516 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260812 03:02:06.263054 26540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:06.264099 26516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 03:02:06.264467 26516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "e520e20c0be1483aa73a80b6b66fef79"
format_stamp: "Formatted at 2026-08-12 03:02:06 on dist-test-slave-zpvx"
server_key: "e2c56666eabd930f564877a213312aa6"
server_key_iv: "e644bddeb3b0fc4b48e1af7e6013f5d5"
server_key_version: "encryptionkey@0"
I20260812 03:02:06.264820 26516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:06.291697 26516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:06.293196 26516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:06.293641 26516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:06.296073 26516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:06.300885 26516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:06.301070 26516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:06.301275 26516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:06.301407 26516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:06.440074 26516 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:33407
I20260812 03:02:06.440186 26734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:33407 every 8 connection(s)
I20260812 03:02:06.442672 26516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:02:06.452134 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 26516
I20260812 03:02:06.452613 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:02:06.453779 10713 external_mini_cluster.cc:1428] Setting key c8ef4c4cc097b9257c625d88391b008c
I20260812 03:02:06.459803 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40241
--builtin_ntp_servers=127.10.118.84:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 03:02:06.461481 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:06.447930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40241 (local address 127.10.118.65:44961)
0812 03:02:06.451177 (+ 3247us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:06.451248 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:06.456347 (+ 5099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:06.457352 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:06.457387 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:06.457559 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:06.458475 (+ 916us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:02:06.458491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:06.458549 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:02:06.458666 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:06.459415 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:06.459446 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:06.459466 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 03:02:06.459732 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 03:02:06.459894 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2590,"thread_start_us":1744,"threads_started":1}
I20260812 03:02:06.461568 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:06.447974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:44961 (local address 127.10.118.126:40241)
0812 03:02:06.449258 (+ 1284us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:06.449276 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:06.451412 (+ 2136us) server_negotiation.cc:407] Connection header received
0812 03:02:06.456733 (+ 5321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:06.456753 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:06.456881 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:06.457028 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:06.458932 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:06.458963 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:06.458993 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:02:06.459063 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:06.459166 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:02:06.459226 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:06.459461 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:06.459716 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 03:02:06.459864 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":654,"threads_started":1}
I20260812 03:02:06.463542 26735 heartbeater.cc:347] Connected to a master server at 127.10.118.126:40241
I20260812 03:02:06.464010 26735 heartbeater.cc:464] Registering TS with master...
I20260812 03:02:06.465094 26735 heartbeater.cc:511] Master 127.10.118.126:40241 requested a full tablet report, sending...
I20260812 03:02:06.467900 26394 ts_manager.cc:194] Registered new tserver with Master: e520e20c0be1483aa73a80b6b66fef79 (127.10.118.65:33407)
W20260812 03:02:06.768442 26743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:06.768996 26743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:06.769630 26743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:06.800277 26743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:06.800571 26743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:06.801479 26743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:02:06.835670 26743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:06.836990 26743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:06.838559 26743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:06.856251 26753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:07.471619 26735 heartbeater.cc:503] Master 127.10.118.126:40241 was elected leader, sending a full tablet report...
W20260812 03:02:06.856343 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:06.857245 26752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:06.858489 26743 server_base.cc:1034] running on GCE node
I20260812 03:02:08.035499 26743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:08.038350 26743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:08.039786 26743 hybrid_clock.cc:648] HybridClock initialized: now 1786503728039728 us; error 81 us; skew 500 ppm
I20260812 03:02:08.040658 26743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:02:08.047222 26743 webserver.cc:460] Webserver started at http://127.10.118.66:46281/ using document root <none> and password file <none>
I20260812 03:02:08.048118 26743 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:08.048307 26743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:08.048813 26743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:08.054368 26743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "27967ec898e24dabb53d870d276cccc6"
format_stamp: "Formatted at 2026-08-12 03:02:08 on dist-test-slave-zpvx"
server_key: "cd96c03f5d56b6c8bc3a318be0a43c99"
server_key_iv: "2b8b97c8372e5c6cf8d3076d977f4663"
server_key_version: "encryptionkey@0"
I20260812 03:02:08.055503 26743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "27967ec898e24dabb53d870d276cccc6"
format_stamp: "Formatted at 2026-08-12 03:02:08 on dist-test-slave-zpvx"
server_key: "cd96c03f5d56b6c8bc3a318be0a43c99"
server_key_iv: "2b8b97c8372e5c6cf8d3076d977f4663"
server_key_version: "encryptionkey@0"
I20260812 03:02:08.062729 26743 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260812 03:02:08.068485 26765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:08.069597 26743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:02:08.069926 26743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "27967ec898e24dabb53d870d276cccc6"
format_stamp: "Formatted at 2026-08-12 03:02:08 on dist-test-slave-zpvx"
server_key: "cd96c03f5d56b6c8bc3a318be0a43c99"
server_key_iv: "2b8b97c8372e5c6cf8d3076d977f4663"
server_key_version: "encryptionkey@0"
I20260812 03:02:08.070312 26743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:08.096151 26743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:08.097645 26743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:08.098124 26743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:08.100664 26743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:08.105548 26743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:08.105748 26743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:08.106060 26743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:08.106240 26743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:08.243167 26743 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:38183
I20260812 03:02:08.243289 26958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:38183 every 8 connection(s)
I20260812 03:02:08.245914 26743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:02:08.246869 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 26743
I20260812 03:02:08.247385 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:02:08.248550 10713 external_mini_cluster.cc:1428] Setting key e7bcea15777c9ce296101ba1ca8e16b3
I20260812 03:02:08.255061 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40241
--builtin_ntp_servers=127.10.118.84:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 03:02:08.270473 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:08.251618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:49817 (local address 127.10.118.126:40241)
0812 03:02:08.253028 (+ 1410us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:08.253075 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:08.258023 (+ 4948us) server_negotiation.cc:407] Connection header received
0812 03:02:08.265676 (+ 7653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:08.265709 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:08.265909 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:08.266124 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:08.267872 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:08.267893 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:08.267913 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:02:08.267965 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:08.268048 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:02:08.268087 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:08.268322 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:08.268657 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 03:02:08.268823 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":536,"threads_started":1}
I20260812 03:02:08.271219 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:08.251971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40241 (local address 127.10.118.66:49817)
0812 03:02:08.257719 (+ 5748us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:08.257822 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:08.265259 (+ 7437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:08.266337 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:08.266361 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:08.266481 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:08.267357 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:02:08.267374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:08.267435 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:02:08.267559 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:08.268266 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:08.268310 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:08.268341 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 03:02:08.268725 (+ 384us) client_negotiation.cc:240] Negotiation successful
0812 03:02:08.268959 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4846,"thread_start_us":3667,"threads_started":1}
I20260812 03:02:08.273262 26960 heartbeater.cc:347] Connected to a master server at 127.10.118.126:40241
I20260812 03:02:08.273672 26960 heartbeater.cc:464] Registering TS with master...
I20260812 03:02:08.274570 26960 heartbeater.cc:511] Master 127.10.118.126:40241 requested a full tablet report, sending...
I20260812 03:02:08.276912 26394 ts_manager.cc:194] Registered new tserver with Master: 27967ec898e24dabb53d870d276cccc6 (127.10.118.66:38183)
W20260812 03:02:08.580996 26968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:08.581566 26968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:08.582175 26968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:08.616199 26968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:08.616597 26968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:08.618124 26968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:02:08.675733 26968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:08.677042 26968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:08.678645 26968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:08.700323 26988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:09.279814 26960 heartbeater.cc:503] Master 127.10.118.126:40241 was elected leader, sending a full tablet report...
W20260812 03:02:08.701316 26982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:08.705461 26985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:08.701764 26968 server_base.cc:1034] running on GCE node
I20260812 03:02:09.876317 26968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:09.878772 26968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:09.880115 26968 hybrid_clock.cc:648] HybridClock initialized: now 1786503729880084 us; error 49 us; skew 500 ppm
I20260812 03:02:09.880964 26968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:02:09.888059 26968 webserver.cc:460] Webserver started at http://127.10.118.67:45977/ using document root <none> and password file <none>
I20260812 03:02:09.889165 26968 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:09.889405 26968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:09.889904 26968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:09.895571 26968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "9416fa465e5f4577b2ccb6620db4dcae"
format_stamp: "Formatted at 2026-08-12 03:02:09 on dist-test-slave-zpvx"
server_key: "d80a2265e682b27013e9d235da93c2b4"
server_key_iv: "1225bafae78f14f639c96cd35e0bcc5d"
server_key_version: "encryptionkey@0"
I20260812 03:02:09.896808 26968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "9416fa465e5f4577b2ccb6620db4dcae"
format_stamp: "Formatted at 2026-08-12 03:02:09 on dist-test-slave-zpvx"
server_key: "d80a2265e682b27013e9d235da93c2b4"
server_key_iv: "1225bafae78f14f639c96cd35e0bcc5d"
server_key_version: "encryptionkey@0"
I20260812 03:02:09.903890 26968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 03:02:09.910465 27000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:09.911597 26968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 03:02:09.911937 26968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "9416fa465e5f4577b2ccb6620db4dcae"
format_stamp: "Formatted at 2026-08-12 03:02:09 on dist-test-slave-zpvx"
server_key: "d80a2265e682b27013e9d235da93c2b4"
server_key_iv: "1225bafae78f14f639c96cd35e0bcc5d"
server_key_version: "encryptionkey@0"
I20260812 03:02:09.912278 26968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:09.937855 26968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:09.939244 26968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:09.939654 26968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:09.942539 26968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:09.946797 26968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:09.946990 26968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:09.947189 26968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:09.947333 26968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:10.084911 26968 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:36277
I20260812 03:02:10.085017 27193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:36277 every 8 connection(s)
I20260812 03:02:10.087730 26968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:02:10.098075 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 26968
I20260812 03:02:10.098569 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:02:10.099651 10713 external_mini_cluster.cc:1428] Setting key f220084fcca8985a39c3f81ff0b9e89e
I20260812 03:02:10.105502 27199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:10.092569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:37487 (local address 127.10.118.126:40241)
0812 03:02:10.093543 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:10.093564 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:10.095803 (+ 2239us) server_negotiation.cc:407] Connection header received
0812 03:02:10.100608 (+ 4805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:10.100650 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:10.100842 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:10.101054 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:10.103078 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:10.103107 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:10.103141 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:02:10.103209 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:10.103327 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:02:10.103366 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:10.103535 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:10.103882 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 03:02:10.104016 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":405,"threads_started":1}
I20260812 03:02:10.105849 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:10.092968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:40241 (local address 127.10.118.67:37487)
0812 03:02:10.095572 (+ 2604us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:10.095637 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:10.100205 (+ 4568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:10.101340 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:10.101375 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:10.101546 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:10.102503 (+ 957us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:02:10.102529 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:10.102615 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:02:10.102778 (+ 163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:10.103541 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:10.103586 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:10.103623 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 03:02:10.103890 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 03:02:10.104053 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2026,"thread_start_us":1264,"threads_started":1}
I20260812 03:02:10.107698 27197 heartbeater.cc:347] Connected to a master server at 127.10.118.126:40241
I20260812 03:02:10.108080 27197 heartbeater.cc:464] Registering TS with master...
I20260812 03:02:10.108843 27197 heartbeater.cc:511] Master 127.10.118.126:40241 requested a full tablet report, sending...
I20260812 03:02:10.110895 26394 ts_manager.cc:194] Registered new tserver with Master: 9416fa465e5f4577b2ccb6620db4dcae (127.10.118.67:36277)
I20260812 03:02:10.119211 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 03:02:10.139240 27199 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:02:10.135411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.10.118.126:40241)
0812 03:02:10.135820 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:10.135838 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:10.137136 (+ 1298us) server_negotiation.cc:407] Connection header received
0812 03:02:10.137463 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:10.137480 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:10.137612 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:10.137747 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:10.138225 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41900: BlockingRecv error: recv got EOF from 127.0.0.1:41900 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 03:02:10.140291 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 26516
I20260812 03:02:10.190547 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 26743
I20260812 03:02:10.224488 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 26968
I20260812 03:02:10.255650 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 26331
2026-08-12T03:02:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (7676 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:02:10 dist-test-slave-zpvx krb5kdc[27212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:02:10 dist-test-slave-zpvx krb5kdc[27212](info): set up 2 sockets
Aug 12 03:02:10 dist-test-slave-zpvx krb5kdc[27212](info): commencing operation
krb5kdc: starting...
W20260812 03:02:11.650048 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:02:11 dist-test-slave-zpvx krb5kdc[27212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:02:11.883014 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44431
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:44431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:02:12.191530 27229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:12.192173 27229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:12.192812 27229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:12.223253 27229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:02:12.223609 27229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:12.223925 27229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:12.224196 27229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:02:12.224474 27229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:02:12.260035 27229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:44431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:12.261312 27229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:12.262449 27229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:12.281292 27240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:12.282807 27229 server_base.cc:1034] running on GCE node
W20260812 03:02:12.281945 27237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:12.281335 27242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:13.478137 27229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:13.480768 27229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:13.482177 27229 hybrid_clock.cc:648] HybridClock initialized: now 1786503733482149 us; error 68 us; skew 500 ppm
I20260812 03:02:13.483086 27229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:02:13 dist-test-slave-zpvx krb5kdc[27212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503733, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:02:13.493003 27229 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:02:13.498241 27229 webserver.cc:460] Webserver started at http://127.10.118.126:34903/ using document root <none> and password file <none>
I20260812 03:02:13.499123 27229 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:13.499302 27229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:13.499702 27229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:13.504945 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "6a2fb15cd5044193ba85a65465bb25bc"
format_stamp: "Formatted at 2026-08-12 03:02:13 on dist-test-slave-zpvx"
server_key: "cb57d2b4a15e2cfcbda6e444381d1e81"
server_key_iv: "3f3168dd4ba545d624366d91a0a5b10a"
server_key_version: "encryptionkey@0"
I20260812 03:02:13.506023 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "6a2fb15cd5044193ba85a65465bb25bc"
format_stamp: "Formatted at 2026-08-12 03:02:13 on dist-test-slave-zpvx"
server_key: "cb57d2b4a15e2cfcbda6e444381d1e81"
server_key_iv: "3f3168dd4ba545d624366d91a0a5b10a"
server_key_version: "encryptionkey@0"
I20260812 03:02:13.513438 27229 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 03:02:13.519598 27253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:13.520714 27229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:02:13.521049 27229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "6a2fb15cd5044193ba85a65465bb25bc"
format_stamp: "Formatted at 2026-08-12 03:02:13 on dist-test-slave-zpvx"
server_key: "cb57d2b4a15e2cfcbda6e444381d1e81"
server_key_iv: "3f3168dd4ba545d624366d91a0a5b10a"
server_key_version: "encryptionkey@0"
I20260812 03:02:13.521426 27229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:13.563355 27229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:13.569306 27229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:13.569782 27229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:13.638963 27229 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:44431
I20260812 03:02:13.639060 27360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:44431 every 8 connection(s)
I20260812 03:02:13.641826 27229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:02:13.647413 27361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:02:13.647462 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 27229
I20260812 03:02:13.648306 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:02:13.650137 10713 external_mini_cluster.cc:1428] Setting key e17df89e8b7406d6978cce6e123734ab
I20260812 03:02:13.675124 27361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc: Bootstrap starting.
Aug 12 03:02:13 dist-test-slave-zpvx krb5kdc[27212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503731, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:02:13.680778 27361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc: Neither blocks nor log segments found. Creating new log.
I20260812 03:02:13.682458 27361 log.cc:824] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc: Log is configured to *not* fsync() on all Append() calls
I20260812 03:02:13.688158 27361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc: No bootstrap required, opened a new log
I20260812 03:02:13.688335 27365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:13.653330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.10.118.126:44431)
0812 03:02:13.656109 (+ 2779us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:13.656136 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:13.656187 (+ 51us) server_negotiation.cc:407] Connection header received
0812 03:02:13.660422 (+ 4235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:13.660477 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:13.660823 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:13.661140 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:13.664774 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:13.666412 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:13.669326 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:13.670151 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:13.670388 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:13.677221 (+ 6833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:13.677246 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:13.677273 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:02:13.677381 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:13.681603 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:13.682704 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:13.682747 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:13.682782 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:13.683026 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:13.683693 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:13.683715 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:13.683729 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:13.684486 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:13.685043 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:13.685758 (+ 715us) server_negotiation.cc:299] Negotiation successful
0812 03:02:13.686418 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2344,"thread_start_us":1858,"threads_started":1}
I20260812 03:02:13.708920 27361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2fb15cd5044193ba85a65465bb25bc" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44431 } }
I20260812 03:02:13.709741 27361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:02:13.710007 27361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2fb15cd5044193ba85a65465bb25bc, State: Initialized, Role: FOLLOWER
I20260812 03:02:13.710611 27361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2fb15cd5044193ba85a65465bb25bc" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44431 } }
I20260812 03:02:13.711097 27361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:02:13.711318 27361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:02:13.711570 27361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:02:13.716220 27361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2fb15cd5044193ba85a65465bb25bc" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44431 } }
I20260812 03:02:13.716945 27361 leader_election.cc:302] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a2fb15cd5044193ba85a65465bb25bc; no voters:
I20260812 03:02:13.718544 27361 leader_election.cc:288] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:02:13.718968 27367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:02:13.721973 27367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [term 1 LEADER]: Becoming Leader. State: Replica: 6a2fb15cd5044193ba85a65465bb25bc, State: Running, Role: LEADER
I20260812 03:02:13.722954 27367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2fb15cd5044193ba85a65465bb25bc" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44431 } }
I20260812 03:02:13.723925 27361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:02:13.733238 27371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a2fb15cd5044193ba85a65465bb25bc. Latest consensus state: current_term: 1 leader_uuid: "6a2fb15cd5044193ba85a65465bb25bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2fb15cd5044193ba85a65465bb25bc" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44431 } } }
I20260812 03:02:13.735493 27371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [sys.catalog]: This master's current role is: LEADER
I20260812 03:02:13.734678 27370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a2fb15cd5044193ba85a65465bb25bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a2fb15cd5044193ba85a65465bb25bc" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44431 } } }
I20260812 03:02:13.737538 27370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc [sys.catalog]: This master's current role is: LEADER
I20260812 03:02:13.739427 27379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:02:13.752311 27379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:02:13.768198 27379 catalog_manager.cc:1347] Generated new cluster ID: b18e36b7c346422f871538d62edc8300
I20260812 03:02:13.768579 27379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:02:13.784816 27379 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:02:13.786577 27379 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:02:13.803232 27379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc: Generated new TSK 0
I20260812 03:02:13.804142 27379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:02:13.880776 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44431
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:02:14.194540 27400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:14.195071 27400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:14.195720 27400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:14.226650 27400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:14.227042 27400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:14.227978 27400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:02:14.265861 27400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:14.267252 27400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:14.268476 27400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:14.292105 27411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:14.292725 27413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:14.293128 27410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:14.294123 27400 server_base.cc:1034] running on GCE node
I20260812 03:02:15.479403 27400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:15.482304 27400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:15.483767 27400 hybrid_clock.cc:648] HybridClock initialized: now 1786503735483721 us; error 48 us; skew 500 ppm
I20260812 03:02:15.484771 27400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:02:15 dist-test-slave-zpvx krb5kdc[27212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503735, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:02:15.491060 27400 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:02:15.495674 27400 webserver.cc:460] Webserver started at http://127.10.118.65:35765/ using document root <none> and password file <none>
I20260812 03:02:15.496691 27400 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:15.496876 27400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:15.497282 27400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:15.502517 27400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5"
format_stamp: "Formatted at 2026-08-12 03:02:15 on dist-test-slave-zpvx"
server_key: "8abee7efcd7a76a0a3d07f9f1b473b0a"
server_key_iv: "c3dd9f0f9fc9543fab04252f1649c160"
server_key_version: "encryptionkey@0"
I20260812 03:02:15.503633 27400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5"
format_stamp: "Formatted at 2026-08-12 03:02:15 on dist-test-slave-zpvx"
server_key: "8abee7efcd7a76a0a3d07f9f1b473b0a"
server_key_iv: "c3dd9f0f9fc9543fab04252f1649c160"
server_key_version: "encryptionkey@0"
I20260812 03:02:15.510888 27400 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260812 03:02:15.516260 27427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:15.517383 27400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 03:02:15.517654 27400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5"
format_stamp: "Formatted at 2026-08-12 03:02:15 on dist-test-slave-zpvx"
server_key: "8abee7efcd7a76a0a3d07f9f1b473b0a"
server_key_iv: "c3dd9f0f9fc9543fab04252f1649c160"
server_key_version: "encryptionkey@0"
I20260812 03:02:15.517984 27400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:15.558362 27400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:15.563603 27400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:15.564003 27400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:15.566509 27400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:15.570428 27400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:15.570613 27400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:15.570807 27400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:15.570938 27400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:15.707397 27400 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:33463
I20260812 03:02:15.707510 27622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:33463 every 8 connection(s)
I20260812 03:02:15.710084 27400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:02:15.720204 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 27400
I20260812 03:02:15.720798 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:02:15.722266 10713 external_mini_cluster.cc:1428] Setting key a094cdc5e7505c8a89fa55b5316d1120
Aug 12 03:02:15 dist-test-slave-zpvx krb5kdc[27212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503735, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:02:15.746794 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:15.714322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:58841 (local address 127.10.118.126:44431)
0812 03:02:15.715394 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:15.715422 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:15.717273 (+ 1851us) server_negotiation.cc:407] Connection header received
0812 03:02:15.723505 (+ 6232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:15.723526 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:15.723700 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:15.723897 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:15.726797 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:15.727722 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:15.729886 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:15.730483 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:15.730763 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:15.735608 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:15.735633 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:15.735649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:02:15.735750 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:15.739639 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:15.741038 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:15.741077 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:15.741104 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:15.741320 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:15.742062 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:15.742081 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:15.742094 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:15.742720 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:15.742962 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:15.744016 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0812 03:02:15.744509 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":447,"threads_started":1}
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
I20260812 03:02:15.747520 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:15.714621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:44431 (local address 127.10.118.65:58841)
0812 03:02:15.717041 (+ 2420us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:15.717108 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:15.723086 (+ 5978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:15.724152 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:15.724187 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:15.725493 (+ 1306us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:15.726520 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:15.726562 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:15.727970 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:15.727989 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:15.729624 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:15.729662 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:15.730820 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:15.730858 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:15.731306 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:15.732211 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:02:15.732275 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:15.735264 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:15.739853 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:02:15.739893 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:02:15.739917 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:02:15.740823 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:02:15.741538 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:02:15.741565 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:02:15.741594 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:02:15.741856 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:02:15.742932 (+ 1076us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:15.742973 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:15.743426 (+ 453us) client_negotiation.cc:769] Sending connection context
0812 03:02:15.743783 (+ 357us) client_negotiation.cc:240] Negotiation successful
0812 03:02:15.744517 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"thread_start_us":1181,"threads_started":1}
I20260812 03:02:15.749531 27624 heartbeater.cc:347] Connected to a master server at 127.10.118.126:44431
I20260812 03:02:15.749994 27624 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:02:15.751112 27624 heartbeater.cc:511] Master 127.10.118.126:44431 requested a full tablet report, sending...
I20260812 03:02:15.753710 27299 ts_manager.cc:194] Registered new tserver with Master: 5997b5f8fd164d2fbd8b8ec15ea224b5 (127.10.118.65:33463)
I20260812 03:02:15.755700 27299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:58841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:02:15.778353 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44431
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:02:16.085808 27632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:16.086364 27632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:16.087015 27632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:16.117736 27632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:16.118121 27632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:16.119055 27632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:02:16.154306 27632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:16.155483 27632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:16.156617 27632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:16.176296 27641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:16.759313 27624 heartbeater.cc:503] Master 127.10.118.126:44431 was elected leader, sending a full tablet report...
I20260812 03:02:16.179250 27632 server_base.cc:1034] running on GCE node
W20260812 03:02:16.176389 27640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:16.177021 27644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:17.361685 27632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:17.364542 27632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:17.365998 27632 hybrid_clock.cc:648] HybridClock initialized: now 1786503737365939 us; error 83 us; skew 500 ppm
I20260812 03:02:17.366775 27632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:02:17 dist-test-slave-zpvx krb5kdc[27212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503737, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:02:17.372788 27632 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:02:17.377604 27632 webserver.cc:460] Webserver started at http://127.10.118.66:43735/ using document root <none> and password file <none>
I20260812 03:02:17.378513 27632 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:17.378733 27632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:17.379206 27632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:17.384987 27632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "47eabcfbc57943988ec8650dde862a00"
format_stamp: "Formatted at 2026-08-12 03:02:17 on dist-test-slave-zpvx"
server_key: "4806a87464ef4597a3016b4505ee10c0"
server_key_iv: "93e60427b891cd8216f77cbfea08b84a"
server_key_version: "encryptionkey@0"
I20260812 03:02:17.386132 27632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "47eabcfbc57943988ec8650dde862a00"
format_stamp: "Formatted at 2026-08-12 03:02:17 on dist-test-slave-zpvx"
server_key: "4806a87464ef4597a3016b4505ee10c0"
server_key_iv: "93e60427b891cd8216f77cbfea08b84a"
server_key_version: "encryptionkey@0"
I20260812 03:02:17.393550 27632 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:02:17.399000 27654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:17.400024 27632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 03:02:17.400347 27632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "47eabcfbc57943988ec8650dde862a00"
format_stamp: "Formatted at 2026-08-12 03:02:17 on dist-test-slave-zpvx"
server_key: "4806a87464ef4597a3016b4505ee10c0"
server_key_iv: "93e60427b891cd8216f77cbfea08b84a"
server_key_version: "encryptionkey@0"
I20260812 03:02:17.400804 27632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:17.436792 27632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:17.442715 27632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:17.443181 27632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:17.445791 27632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:17.449762 27632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:17.449998 27632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:17.450264 27632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:17.450441 27632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:17.585870 27632 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:44799
I20260812 03:02:17.586068 27845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:44799 every 8 connection(s)
I20260812 03:02:17.588486 27632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:02:17.592015 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 27632
I20260812 03:02:17.592559 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:02:17.594128 10713 external_mini_cluster.cc:1428] Setting key 622c825e4ec56fbd892b416f2fc43aea
Aug 12 03:02:17 dist-test-slave-zpvx krb5kdc[27212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503737, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:02:17.628592 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:17.593166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:52595 (local address 127.10.118.126:44431)
0812 03:02:17.597183 (+ 4017us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:17.597215 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:17.599977 (+ 2762us) server_negotiation.cc:407] Connection header received
0812 03:02:17.604737 (+ 4760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:17.604769 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:17.604945 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:17.605130 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:17.607486 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:17.609029 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:17.611162 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:17.611758 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:17.612062 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:17.617580 (+ 5518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:17.617604 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:17.617620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:02:17.617731 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:17.621808 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:17.623409 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:17.623430 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:17.623449 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:17.623637 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:17.624618 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:17.624638 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:17.624652 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:17.625222 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:17.625406 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:17.626722 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0812 03:02:17.627066 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3406,"thread_start_us":3109,"threads_started":1}
I20260812 03:02:17.630505 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:17.593642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:44431 (local address 127.10.118.66:52595)
0812 03:02:17.599749 (+ 6107us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:17.599818 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:17.604320 (+ 4502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:17.605417 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:17.605457 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:17.606295 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:17.607181 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:17.607230 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:17.609349 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:17.609371 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:17.610910 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:17.610947 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:17.612114 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:17.612150 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:17.612638 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:17.613605 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:02:17.613668 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:17.617229 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:17.622029 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:02:17.622066 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:02:17.622099 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:02:17.623205 (+ 1106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:02:17.623881 (+ 676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:02:17.623912 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:02:17.623936 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:02:17.624431 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:02:17.625415 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:17.625447 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:17.626023 (+ 576us) client_negotiation.cc:769] Sending connection context
0812 03:02:17.626452 (+ 429us) client_negotiation.cc:240] Negotiation successful
0812 03:02:17.627173 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5539,"thread_start_us":4642,"threads_started":1}
I20260812 03:02:17.632682 27846 heartbeater.cc:347] Connected to a master server at 127.10.118.126:44431
I20260812 03:02:17.633095 27846 heartbeater.cc:464] Registering TS with master...
I20260812 03:02:17.633999 27846 heartbeater.cc:511] Master 127.10.118.126:44431 requested a full tablet report, sending...
I20260812 03:02:17.636260 27299 ts_manager.cc:194] Registered new tserver with Master: 47eabcfbc57943988ec8650dde862a00 (127.10.118.66:44799)
I20260812 03:02:17.638545 27299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:52595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:02:17.658994 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44431
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:02:17.965765 27854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:02:17.966310 27854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:02:17.966964 27854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:02:17.997107 27854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:02:17.997496 27854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:02:17.998449 27854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:02:18.034116 27854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:02:18.035318 27854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:02:18.036360 27854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:02:18.056082 27864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:18.642976 27846 heartbeater.cc:503] Master 127.10.118.126:44431 was elected leader, sending a full tablet report...
W20260812 03:02:18.056121 27863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:02:18.057174 27871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:02:18.057897 27854 server_base.cc:1034] running on GCE node
I20260812 03:02:19.254546 27854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:02:19.256971 27854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:02:19.258345 27854 hybrid_clock.cc:648] HybridClock initialized: now 1786503739258313 us; error 52 us; skew 500 ppm
I20260812 03:02:19.259214 27854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:02:19 dist-test-slave-zpvx krb5kdc[27212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503739, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:02:19.265656 27854 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:02:19.271306 27854 webserver.cc:460] Webserver started at http://127.10.118.67:40219/ using document root <none> and password file <none>
I20260812 03:02:19.272341 27854 fs_manager.cc:359] Metadata directory not provided
I20260812 03:02:19.272732 27854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:02:19.273288 27854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:02:19.278546 27854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "277c442907af4de0b573a1d267e912d1"
format_stamp: "Formatted at 2026-08-12 03:02:19 on dist-test-slave-zpvx"
server_key: "116286e297f60497d0da854f0d6bbad4"
server_key_iv: "c9d29f7392530466d363b24768864a09"
server_key_version: "encryptionkey@0"
I20260812 03:02:19.279713 27854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "277c442907af4de0b573a1d267e912d1"
format_stamp: "Formatted at 2026-08-12 03:02:19 on dist-test-slave-zpvx"
server_key: "116286e297f60497d0da854f0d6bbad4"
server_key_iv: "c9d29f7392530466d363b24768864a09"
server_key_version: "encryptionkey@0"
I20260812 03:02:19.286991 27854 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.007s
I20260812 03:02:19.292639 27889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:19.293677 27854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:02:19.294009 27854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "277c442907af4de0b573a1d267e912d1"
format_stamp: "Formatted at 2026-08-12 03:02:19 on dist-test-slave-zpvx"
server_key: "116286e297f60497d0da854f0d6bbad4"
server_key_iv: "c9d29f7392530466d363b24768864a09"
server_key_version: "encryptionkey@0"
I20260812 03:02:19.294394 27854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:02:19.355579 27854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:02:19.364037 27854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:02:19.364646 27854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:02:19.368153 27854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:02:19.374327 27854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:02:19.374584 27854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:19.374866 27854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:02:19.375064 27854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:02:19.529896 27854 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:38863
I20260812 03:02:19.530047 28064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:38863 every 8 connection(s)
I20260812 03:02:19.532723 27854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:02:19.533587 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 27854
I20260812 03:02:19.534143 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:02:19.535853 10713 external_mini_cluster.cc:1428] Setting key 3b48acc8bddc2ebdfaf0af65274190fe
Aug 12 03:02:19 dist-test-slave-zpvx krb5kdc[27212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503739, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:02:19.569648 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.537955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:55079 (local address 127.10.118.126:44431)
0812 03:02:19.540032 (+ 2077us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:19.540082 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:19.542233 (+ 2151us) server_negotiation.cc:407] Connection header received
0812 03:02:19.546677 (+ 4444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:19.546696 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:19.546864 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:19.547008 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:19.549204 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.550183 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.552221 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.553125 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.553389 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.558999 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:19.559026 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:19.559051 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:02:19.559198 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:19.563167 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:19.564295 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:19.564315 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:19.564331 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:19.564594 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:19.565273 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:19.565295 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:19.565309 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:19.565830 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:19.566012 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:19.567065 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0812 03:02:19.567524 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1716,"thread_start_us":1407,"threads_started":1}
I20260812 03:02:19.569837 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.538274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:44431 (local address 127.10.118.67:55079)
0812 03:02:19.541921 (+ 3647us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:19.542014 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:19.546369 (+ 4355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:19.547229 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:19.547252 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:19.548050 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:02:19.548944 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.548987 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.550395 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.550414 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.551971 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.552012 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.553448 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.553474 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.553903 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.554991 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:02:19.555090 (+ 99us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:02:19.558667 (+ 3577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:02:19.563355 (+ 4688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:02:19.563378 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:02:19.563406 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:02:19.564116 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:02:19.564760 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:02:19.564778 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:02:19.564792 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:02:19.565058 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:02:19.566018 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:02:19.566047 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:02:19.566452 (+ 405us) client_negotiation.cc:769] Sending connection context
0812 03:02:19.566757 (+ 305us) client_negotiation.cc:240] Negotiation successful
0812 03:02:19.567215 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3007,"thread_start_us":2191,"threads_started":1}
I20260812 03:02:19.571818 28066 heartbeater.cc:347] Connected to a master server at 127.10.118.126:44431
I20260812 03:02:19.572181 28066 heartbeater.cc:464] Registering TS with master...
I20260812 03:02:19.573179 28066 heartbeater.cc:511] Master 127.10.118.126:44431 requested a full tablet report, sending...
I20260812 03:02:19.575254 27299 ts_manager.cc:194] Registered new tserver with Master: 277c442907af4de0b573a1d267e912d1 (127.10.118.67:38863)
I20260812 03:02:19.576715 27299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:55079
I20260812 03:02:19.585183 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:02:19.626817 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.605181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.10.118.126:44431)
0812 03:02:19.605594 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:19.605613 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:19.606639 (+ 1026us) server_negotiation.cc:407] Connection header received
0812 03:02:19.607074 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:19.607099 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:19.607299 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:19.607477 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:02:19.609845 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.610759 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.612720 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.613326 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.613601 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.617370 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:02:19.617393 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:02:19.617409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:02:19.617532 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:02:19.621169 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:19.622260 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:19.622281 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:19.622295 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:19.622493 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:02:19.623144 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:02:19.623164 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:02:19.623180 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:02:19.623693 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:02:19.624191 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:19.624924 (+ 733us) server_negotiation.cc:299] Negotiation successful
0812 03:02:19.625368 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 03:02:19.633226 27299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:02:19.665385 27299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:02:19.720832 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.694707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.10.118.65:33463)
0812 03:02:19.698036 (+ 3329us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:19.698089 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:19.700960 (+ 2871us) server_negotiation.cc:407] Connection header received
0812 03:02:19.701660 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:19.701697 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:19.702070 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:19.702274 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:02:19.704535 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.707181 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.712035 (+ 4854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.714650 (+ 2615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.714996 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.715622 (+ 626us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:19.718808 (+ 3186us) server_negotiation.cc:299] Negotiation successful
0812 03:02:19.719117 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2712,"thread_start_us":2301,"threads_started":1}
I20260812 03:02:19.723820 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.694184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33463 (local address 127.0.0.1:36932)
0812 03:02:19.700668 (+ 6484us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:19.700777 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:19.701290 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:19.703003 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:19.703042 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:19.703190 (+ 148us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:02:19.704040 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.704121 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.707652 (+ 3531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.707695 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.711710 (+ 4015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.711766 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.716040 (+ 4274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.716071 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.716715 (+ 644us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.716950 (+ 235us) client_negotiation.cc:769] Sending connection context
0812 03:02:19.721584 (+ 4634us) client_negotiation.cc:240] Negotiation successful
0812 03:02:19.721978 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5066,"mutex_wait_us":78,"thread_start_us":6662,"threads_started":1}
I20260812 03:02:19.728927 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.698196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.10.118.66:44799)
0812 03:02:19.700590 (+ 2394us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:19.700623 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:19.702001 (+ 1378us) server_negotiation.cc:407] Connection header received
0812 03:02:19.702465 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:19.702496 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:19.702923 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:19.703135 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:02:19.705331 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.707399 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.722430 (+ 15031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.724617 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.724890 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.725393 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:19.727007 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0812 03:02:19.727316 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1863,"thread_start_us":1411,"threads_started":1}
I20260812 03:02:19.729266 27507 tablet_service.cc:1467] Processing CreateTablet for tablet 10fd3b3dc8cd4411a7524257cc73ce0c (DEFAULT_TABLE table=test-table [id=cb3122b7772d4486bb0488b329195e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:02:19.731369 27507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10fd3b3dc8cd4411a7524257cc73ce0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:02:19.732017 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.696961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:44799 (local address 127.0.0.1:41902)
0812 03:02:19.701612 (+ 4651us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:19.701847 (+ 235us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:19.702131 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:19.703846 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:19.703887 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:19.704045 (+ 158us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:02:19.704969 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.705021 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.707769 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.707800 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.722089 (+ 14289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.722152 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.725811 (+ 3659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.725845 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.726501 (+ 656us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.726695 (+ 194us) client_negotiation.cc:769] Sending connection context
0812 03:02:19.729807 (+ 3112us) client_negotiation.cc:240] Negotiation successful
0812 03:02:19.730133 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4184,"mutex_wait_us":10856,"thread_start_us":1479,"threads_started":1}
I20260812 03:02:19.734906 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.702350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:38863 (local address 127.0.0.1:34428)
0812 03:02:19.719280 (+ 16930us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:19.719342 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:19.719592 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:19.721147 (+ 1555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:19.721180 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:19.721277 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:02:19.722410 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.722455 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.725303 (+ 2848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.725339 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.728333 (+ 2994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.728413 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.731849 (+ 3436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.731881 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.732338 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.732501 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 03:02:19.732676 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 03:02:19.732989 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16150,"mutex_wait_us":310,"thread_start_us":15864,"threads_started":1}
I20260812 03:02:19.736009 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.702160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.10.118.67:38863)
0812 03:02:19.718568 (+ 16408us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:19.718602 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:19.719812 (+ 1210us) server_negotiation.cc:407] Connection header received
0812 03:02:19.720088 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:19.720126 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:19.720479 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:19.720690 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:02:19.722792 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.724970 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.728743 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.730798 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.731115 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.731570 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:19.733957 (+ 2387us) server_negotiation.cc:299] Negotiation successful
0812 03:02:19.734278 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15876,"thread_start_us":15449,"threads_started":1}
I20260812 03:02:19.736860 27731 tablet_service.cc:1467] Processing CreateTablet for tablet 10fd3b3dc8cd4411a7524257cc73ce0c (DEFAULT_TABLE table=test-table [id=cb3122b7772d4486bb0488b329195e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:02:19.739318 27731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10fd3b3dc8cd4411a7524257cc73ce0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:02:19.740734 27960 tablet_service.cc:1467] Processing CreateTablet for tablet 10fd3b3dc8cd4411a7524257cc73ce0c (DEFAULT_TABLE table=test-table [id=cb3122b7772d4486bb0488b329195e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:02:19.742708 27960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10fd3b3dc8cd4411a7524257cc73ce0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:02:19.758705 28102 tablet_bootstrap.cc:492] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5: Bootstrap starting.
I20260812 03:02:19.764433 28102 tablet_bootstrap.cc:654] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5: Neither blocks nor log segments found. Creating new log.
I20260812 03:02:19.766691 28102 log.cc:824] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5: Log is configured to *not* fsync() on all Append() calls
I20260812 03:02:19.773646 28102 tablet_bootstrap.cc:492] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5: No bootstrap required, opened a new log
I20260812 03:02:19.774181 28102 ts_tablet_manager.cc:1397] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20260812 03:02:19.776955 28109 tablet_bootstrap.cc:492] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Bootstrap starting.
I20260812 03:02:19.778494 28105 tablet_bootstrap.cc:492] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Bootstrap starting.
I20260812 03:02:19.782258 28109 tablet_bootstrap.cc:654] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Neither blocks nor log segments found. Creating new log.
I20260812 03:02:19.784008 28109 log.cc:824] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Log is configured to *not* fsync() on all Append() calls
I20260812 03:02:19.785626 28105 tablet_bootstrap.cc:654] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Neither blocks nor log segments found. Creating new log.
I20260812 03:02:19.788012 28105 log.cc:824] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Log is configured to *not* fsync() on all Append() calls
I20260812 03:02:19.791811 28109 tablet_bootstrap.cc:492] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: No bootstrap required, opened a new log
I20260812 03:02:19.792343 28109 ts_tablet_manager.cc:1397] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.001s
I20260812 03:02:19.795018 28105 tablet_bootstrap.cc:492] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: No bootstrap required, opened a new log
I20260812 03:02:19.795504 28105 ts_tablet_manager.cc:1397] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20260812 03:02:19.806155 28102 raft_consensus.cc:348] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.807104 28102 raft_consensus.cc:374] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:02:19.807432 28102 raft_consensus.cc:729] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5997b5f8fd164d2fbd8b8ec15ea224b5, State: Initialized, Role: FOLLOWER
I20260812 03:02:19.808321 28102 consensus_queue.cc:260] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.813334 28102 ts_tablet_manager.cc:1428] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5: Time spent starting tablet: real 0.039s user 0.032s sys 0.006s
I20260812 03:02:19.815984 28109 raft_consensus.cc:348] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.816741 28109 raft_consensus.cc:374] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:02:19.817071 28109 raft_consensus.cc:729] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eabcfbc57943988ec8650dde862a00, State: Initialized, Role: FOLLOWER
I20260812 03:02:19.817860 28109 consensus_queue.cc:260] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.822841 28109 ts_tablet_manager.cc:1428] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Time spent starting tablet: real 0.030s user 0.021s sys 0.005s
I20260812 03:02:19.825465 28105 raft_consensus.cc:348] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.826382 28105 raft_consensus.cc:374] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:02:19.826721 28105 raft_consensus.cc:729] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277c442907af4de0b573a1d267e912d1, State: Initialized, Role: FOLLOWER
I20260812 03:02:19.827574 28105 consensus_queue.cc:260] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.830210 28066 heartbeater.cc:503] Master 127.10.118.126:44431 was elected leader, sending a full tablet report...
I20260812 03:02:19.831027 28105 ts_tablet_manager.cc:1428] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Time spent starting tablet: real 0.035s user 0.037s sys 0.000s
W20260812 03:02:19.850370 27847 tablet.cc:2369] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:02:19.930223 28116 raft_consensus.cc:482] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:02:19.930671 28116 raft_consensus.cc:504] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.933401 28116 leader_election.cc:288] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 277c442907af4de0b573a1d267e912d1 (127.10.118.67:38863), 47eabcfbc57943988ec8650dde862a00 (127.10.118.66:44799)
I20260812 03:02:19.948717 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.933848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:47833 (local address 127.10.118.67:38863)
0812 03:02:19.934582 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:19.934615 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:19.937393 (+ 2778us) server_negotiation.cc:407] Connection header received
0812 03:02:19.937845 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:19.937874 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:19.938104 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:19.938267 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:02:19.940042 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.940961 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.944263 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.945314 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.945623 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.945916 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:19.946953 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0812 03:02:19.947237 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260812 03:02:19.948865 28097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.934454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:49079 (local address 127.10.118.66:44799)
0812 03:02:19.935118 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:19.935149 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:19.937328 (+ 2179us) server_negotiation.cc:407] Connection header received
0812 03:02:19.937777 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:19.937810 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:19.938082 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:19.938268 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:02:19.939576 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.940907 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.944085 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.945363 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.945847 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.946196 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:19.946921 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 03:02:19.947236 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260812 03:02:19.949565 28123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.933981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:44799 (local address 127.10.118.65:49079)
0812 03:02:19.937133 (+ 3152us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:19.937177 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:19.937492 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:19.938330 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:19.938359 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:19.938457 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:02:19.939287 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.939323 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.941185 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.941224 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.943801 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.943840 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.945792 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.945827 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.946541 (+ 714us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.946663 (+ 122us) client_negotiation.cc:769] Sending connection context
0812 03:02:19.946786 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 03:02:19.947201 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2789,"mutex_wait_us":574,"spinlock_wait_cycles":50048,"thread_start_us":2478,"threads_started":1}
I20260812 03:02:19.949940 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:19.933697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:38863 (local address 127.10.118.65:47833)
0812 03:02:19.937167 (+ 3470us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:19.937214 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 03:02:19.937538 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:02:19.938330 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:02:19.938372 (+ 42us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:02:19.938598 (+ 226us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:02:19.939689 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.939748 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.941250 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.941278 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.943959 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:02:19.944019 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:02:19.945695 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:02:19.945731 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:02:19.946567 (+ 836us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:19.946697 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 03:02:19.946889 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 03:02:19.947330 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3000,"mutex_wait_us":576,"spinlock_wait_cycles":40960,"thread_start_us":2254,"threads_started":1}
I20260812 03:02:19.951504 27776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd3b3dc8cd4411a7524257cc73ce0c" candidate_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabcfbc57943988ec8650dde862a00" is_pre_election: true
I20260812 03:02:19.951650 27997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd3b3dc8cd4411a7524257cc73ce0c" candidate_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277c442907af4de0b573a1d267e912d1" is_pre_election: true
I20260812 03:02:19.952533 27776 raft_consensus.cc:2454] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5997b5f8fd164d2fbd8b8ec15ea224b5 in term 0.
I20260812 03:02:19.952631 27997 raft_consensus.cc:2454] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5997b5f8fd164d2fbd8b8ec15ea224b5 in term 0.
I20260812 03:02:19.953897 27433 leader_election.cc:302] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277c442907af4de0b573a1d267e912d1, 5997b5f8fd164d2fbd8b8ec15ea224b5; no voters:
I20260812 03:02:19.954793 28116 raft_consensus.cc:2781] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:02:19.955049 28116 raft_consensus.cc:482] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:02:19.955346 28116 raft_consensus.cc:3037] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:02:19.959990 28116 raft_consensus.cc:504] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.961462 28116 leader_election.cc:288] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [CANDIDATE]: Term 1 election: Requested vote from peers 277c442907af4de0b573a1d267e912d1 (127.10.118.67:38863), 47eabcfbc57943988ec8650dde862a00 (127.10.118.66:44799)
I20260812 03:02:19.962283 27997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd3b3dc8cd4411a7524257cc73ce0c" candidate_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277c442907af4de0b573a1d267e912d1"
I20260812 03:02:19.962373 27776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10fd3b3dc8cd4411a7524257cc73ce0c" candidate_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47eabcfbc57943988ec8650dde862a00"
I20260812 03:02:19.962756 27997 raft_consensus.cc:3037] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:02:19.962848 27776 raft_consensus.cc:3037] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:02:19.967480 27776 raft_consensus.cc:2454] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5997b5f8fd164d2fbd8b8ec15ea224b5 in term 1.
I20260812 03:02:19.967478 27997 raft_consensus.cc:2454] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5997b5f8fd164d2fbd8b8ec15ea224b5 in term 1.
I20260812 03:02:19.968278 27433 leader_election.cc:302] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277c442907af4de0b573a1d267e912d1, 5997b5f8fd164d2fbd8b8ec15ea224b5; no voters:
I20260812 03:02:19.968951 28116 raft_consensus.cc:2781] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 1 FOLLOWER]: Leader election won for term 1
W20260812 03:02:19.969575 27625 tablet.cc:2369] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:02:19.971207 28116 raft_consensus.cc:686] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [term 1 LEADER]: Becoming Leader. State: Replica: 5997b5f8fd164d2fbd8b8ec15ea224b5, State: Running, Role: LEADER
I20260812 03:02:19.972044 28116 consensus_queue.cc:237] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } }
I20260812 03:02:19.983947 27296 catalog_manager.cc:5495] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5997b5f8fd164d2fbd8b8ec15ea224b5 (127.10.118.65). New cstate: current_term: 1 leader_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5997b5f8fd164d2fbd8b8ec15ea224b5" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 33463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 } health_report { overall_health: UNKNOWN } } }
I20260812 03:02:20.031512 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:20.023004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.10.118.65:33463)
0812 03:02:20.023455 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0812 03:02:20.023471 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 03:02:20.023511 (+ 40us) server_negotiation.cc:407] Connection header received
0812 03:02:20.023851 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:02:20.023869 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:02:20.024107 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:02:20.024335 (+ 228us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:02:20.025029 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:20.025960 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:20.027731 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:02:20.028406 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:02:20.028619 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:02:20.029357 (+ 738us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:02:20.029867 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:02:20.030023 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 03:02:20.030221 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 03:02:20.030358 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260812 03:02:20.039768 28067 tablet.cc:2369] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:02:20.045360 27776 raft_consensus.cc:1261] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 1 FOLLOWER]: Refusing update from remote peer 5997b5f8fd164d2fbd8b8ec15ea224b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:02:20.045359 27997 raft_consensus.cc:1261] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 1 FOLLOWER]: Refusing update from remote peer 5997b5f8fd164d2fbd8b8ec15ea224b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:02:20.046955 28116 consensus_queue.cc:1035] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "277c442907af4de0b573a1d267e912d1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 38863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:02:20.047749 28125 consensus_queue.cc:1035] T 10fd3b3dc8cd4411a7524257cc73ce0c P 5997b5f8fd164d2fbd8b8ec15ea224b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47eabcfbc57943988ec8650dde862a00" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:02:20.082619 28132 mvcc.cc:205] Tried to move back new op lower bound from 7317519319208062976 to 7317519318932918272. Current Snapshot: MvccSnapshot[applied={T|T < 7317519319208062976}]
I20260812 03:02:20.095502 27296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:02:20.096030 27296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:02:20.105235 27296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a2fb15cd5044193ba85a65465bb25bc: Sending DeleteTablet for 3 replicas of tablet 10fd3b3dc8cd4411a7524257cc73ce0c
I20260812 03:02:20.107103 27507 tablet_service.cc:1514] Processing DeleteTablet for tablet 10fd3b3dc8cd4411a7524257cc73ce0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:02:20 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:36932
I20260812 03:02:20.107451 27960 tablet_service.cc:1514] Processing DeleteTablet for tablet 10fd3b3dc8cd4411a7524257cc73ce0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:02:20 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:34428
I20260812 03:02:20.107599 27731 tablet_service.cc:1514] Processing DeleteTablet for tablet 10fd3b3dc8cd4411a7524257cc73ce0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:02:20 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:41902
I20260812 03:02:20.118078 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 27400
I20260812 03:02:20.128242 28153 tablet_replica.cc:330] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: stopping tablet replica
I20260812 03:02:20.129650 28153 raft_consensus.cc:2229] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:02:20.130040 28154 tablet_replica.cc:330] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: stopping tablet replica
I20260812 03:02:20.130317 28153 raft_consensus.cc:2258] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:02:20.130950 28154 raft_consensus.cc:2229] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:02:20.131839 28154 raft_consensus.cc:2258] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:02:20.133737 28153 ts_tablet_manager.cc:1905] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:02:20.135397 28154 ts_tablet_manager.cc:1905] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:02:20.144954 28153 ts_tablet_manager.cc:1918] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:02:20.145332 28153 log.cc:1196] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/10fd3b3dc8cd4411a7524257cc73ce0c
I20260812 03:02:20.145975 28153 ts_tablet_manager.cc:1939] T 10fd3b3dc8cd4411a7524257cc73ce0c P 277c442907af4de0b573a1d267e912d1: Deleting consensus metadata
I20260812 03:02:20.148478 27259 catalog_manager.cc:4841] TS 277c442907af4de0b573a1d267e912d1 (127.10.118.67:38863): tablet 10fd3b3dc8cd4411a7524257cc73ce0c (table test-table [id=cb3122b7772d4486bb0488b329195e94]) successfully deleted
I20260812 03:02:20.153069 28154 ts_tablet_manager.cc:1918] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:02:20.153539 28154 log.cc:1196] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/10fd3b3dc8cd4411a7524257cc73ce0c
I20260812 03:02:20.154404 28154 ts_tablet_manager.cc:1939] T 10fd3b3dc8cd4411a7524257cc73ce0c P 47eabcfbc57943988ec8650dde862a00: Deleting consensus metadata
I20260812 03:02:20.157749 27255 catalog_manager.cc:4841] TS 47eabcfbc57943988ec8650dde862a00 (127.10.118.66:44799): tablet 10fd3b3dc8cd4411a7524257cc73ce0c (table test-table [id=cb3122b7772d4486bb0488b329195e94]) successfully deleted
W20260812 03:02:20.167109 27259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:33463 (error 108)
I20260812 03:02:20.167443 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 27632
I20260812 03:02:20.181022 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:20.179327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33463 (local address 127.0.0.1:36952)
0812 03:02:20.179946 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:20.180239 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260812 03:02:20.181584 27259 catalog_manager.cc:4584] TS 5997b5f8fd164d2fbd8b8ec15ea224b5 (127.10.118.65:33463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10fd3b3dc8cd4411a7524257cc73ce0c: Network error: Client connection negotiation failed: client connection to 127.10.118.65:33463: connect: Connection refused (error 111)
I20260812 03:02:20.205974 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 27854
I20260812 03:02:20.236804 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:20.235175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33463 (local address 127.0.0.1:36968)
0812 03:02:20.235840 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:20.236086 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 03:02:20.239595 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:02:20.238163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:33463 (local address 127.0.0.1:36972)
0812 03:02:20.238758 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 03:02:20.238988 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:33463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 03:02:20.247862 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 27229
2026-08-12T03:02:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (9993 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:02:20 dist-test-slave-zpvx krb5kdc[28161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:02:20 dist-test-slave-zpvx krb5kdc[28161](info): set up 2 sockets
Aug 12 03:02:20 dist-test-slave-zpvx krb5kdc[28161](info): commencing operation
krb5kdc: starting...
W20260812 03:02:21.642795 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:02:21 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:02:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/rangeradmin_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/rangeradmin_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/rangerlookup_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/rangerlookup_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
I20260812 03:02:21.963876 10713 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgres/pg_ctl -D /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/postgres -l logfile start
2026-08-12 03:02:24.519 UTC [28201] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 03:02:24.519 UTC [28201] LOG: listening on IPv4 address "127.10.118.84", port 34927
2026-08-12 03:02:24.523 UTC [28201] LOG: listening on Unix socket "/tmp/.s.PGSQL.34927"
2026-08-12 03:02:24.547 UTC [28204] LOG: database system was shut down at 2026-08-12 03:02:23 UTC
2026-08-12 03:02:24.551 UTC [28201] LOG: database system is ready to accept connections
I20260812 03:02:25.793371 10713 mini_postgres.cc:96] Postgres bound to 34927
2026-08-12 03:02:25.811 UTC [28222] FATAL: database "slave" does not exist
127.10.118.84:34927 - accepting connections
I20260812 03:02:25.812830 10713 mini_ranger.cc:161] Starting Ranger...
I20260812 03:02:25.838292 10713 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 03:02:26.345592 10713 mini_ranger.cc:87] Created ranger Postgres database
I20260812 03:02:26.345840 10713 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-admin
2026-08-12 03:02:26,659 [I] DB FLAVOR :POSTGRES
2026-08-12 03:02:26,659 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 03:02:26,659 [I] Checking connection..
2026-08-12 03:02:26,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 03:02:26,915 [I] Checking connection passed.
2026-08-12 03:02:26,915 [I] --------- Verifying version history table ---------
2026-08-12 03:02:26,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 03:02:27,168 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 03:02:27,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 03:02:27,420 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 03:02:27,421 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 03:02:27,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 03:02:27.663 UTC [28321] WARNING: there is no transaction in progress
2026-08-12 03:02:27,672 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 03:02:27,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 03:02:27,951 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 03:02:27,951 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 03:02:27,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 03:02:28,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 03:02:28,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpvx.c.gcp-upstream.internal','N') ;"
2026-08-12 03:02:28,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 03:02:28,959 [I] Table x_portal_user does not exist in database ranger
2026-08-12 03:02:28,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 03:02:29,214 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 03:02:29,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 03:02:29,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 03:02:29,726 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 03:02:29,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 03:02:30.708 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.724 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.743 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.758 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.773 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.790 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.862 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.869 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.883 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.894 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.908 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.921 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.931 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.937 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.945 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:30.951 UTC [28527] WARNING: there is no transaction in progress
2026-08-12 03:02:31,589 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 03:02:31,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 03:02:31,857 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 03:02:31,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 03:02:32,102 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 03:02:32,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 03:02:32,364 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 03:02:32,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 03:02:32,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 03:02:32,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpvx.c.gcp-upstream.internal';"
2026-08-12 03:02:33,118 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 03:02:33,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 03:02:33,348 [I] DB_PATCHES have already been applied
I20260812 03:02:33.354859 10713 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx4bzWQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 03:02:33.355154 10713 mini_ranger.cc:193] Using host: 127.10.118.84
I20260812 03:02:33.372129 10713 mini_ranger.cc:227] Ranger admin URL: http://127.10.118.84:50811
Aug 12, 2026 3:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 3:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 3:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 3:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 3:02:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 3:02:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 3:02:34 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 03:02:34.793244 10713 mini_ranger.cc:160] Time spent starting Ranger: real 8.980s user 0.018s sys 0.031s
Aug 12, 2026 3:02:55 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 3:02:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 3:02:55 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 3:02:55 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 3:02:56 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 03:02:56.743341 10713 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/rangerkms_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/rangerkms_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.118.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.118.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 03:02:56 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503776, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 03:02:56.911480 10713 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 03:02:56.931734 10713 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 03:02:57.829528 10713 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 03:02:57.829921 10713 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms
I20260812 03:02:57.830148 10713 mini_ranger_kms.cc:227] Using postgres at 127.10.118.84:34927
2026-08-12 03:02:58,858 [I] DB FLAVOR :POSTGRES
2026-08-12 03:02:58,858 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 03:02:58,858 [I] Checking connection
2026-08-12 03:02:58,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 03:02:59,118 [I] connection success
2026-08-12 03:02:59,119 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 03:02:59,119 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 03:02:59,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 03:02:59,359 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 03:02:59,359 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 03:02:59,359 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 03:02:59,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:34927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 03:02:59,618 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 03:03:00.104318 10713 mini_ranger_kms.cc:310] Created kms service
I20260812 03:03:00.286351 10713 mini_ranger_kms.cc:326] Created kudu user
I20260812 03:03:00.334347 10713 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 03:03:00 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503776, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:03:00.793797 10713 mini_ranger_kms.cc:384] Added ranger policy
I20260812 03:03:00.794356 10713 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 03:03:00.794559 10713 mini_ranger_kms.cc:240] Using host: 127.10.118.84
I20260812 03:03:00.812731 10713 mini_ranger_kms.cc:276] Ranger KMS PID: 28859
I20260812 03:03:00.813122 10713 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.118.84:52585
Aug 12, 2026 3:03:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 3:03:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 3:03:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 3:03:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 3:03:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 3:03:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 03:03:03 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786503783, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.118.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 03:03:03 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503783, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.118.84@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:03:03.692648 10713 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.781s user 0.348s sys 0.697s
I20260812 03:03:03.693321 10713 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 03:03:03.693910 10713 mini_ranger_kms.cc:401] 127.10.118.84:52585/kms/v1/keys
Aug 12 03:03:03 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503783, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.118.84@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
Aug 12 03:03:06 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:03:06.223006 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:37301
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:37301
--ranger_config_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:52585/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:03:06.535555 28936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:03:06.536214 28936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:03:06.536846 28936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:03:06.566596 28936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:03:06.566987 28936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:03:06.567312 28936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:03:06.567571 28936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:03:06.567828 28936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:03:06.603106 28936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.118.84:52585/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:37301
--ranger_config_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:37301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:03:06.604310 28936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:03:06.605397 28936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:03:06.624042 28949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:03:06.624042 28950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:03:06.624855 28952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:03:06.625908 28936 server_base.cc:1034] running on GCE node
I20260812 03:03:07.819742 28936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:03:07.822636 28936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:03:07.824028 28936 hybrid_clock.cc:648] HybridClock initialized: now 1786503787823998 us; error 47 us; skew 500 ppm
I20260812 03:03:07.824910 28936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:03:07 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503787, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:03:07.834583 28936 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:03:07.839917 28936 webserver.cc:460] Webserver started at http://127.10.118.126:37995/ using document root <none> and password file <none>
I20260812 03:03:07.841017 28936 fs_manager.cc:359] Metadata directory not provided
I20260812 03:03:07.841248 28936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:03:07.841727 28936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:03:07 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503787, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:03:07.905624 28936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "057bcdd9d9134ac0b1b11db70b8f9c98"
format_stamp: "Formatted at 2026-08-12 03:03:07 on dist-test-slave-zpvx"
server_key: "97763d4a855865c40512cf8c72fa8f5f"
server_key_iv: "c81866947aac8dab758bd7c26b6f25db"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:07.906945 28936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "057bcdd9d9134ac0b1b11db70b8f9c98"
format_stamp: "Formatted at 2026-08-12 03:03:07 on dist-test-slave-zpvx"
server_key: "97763d4a855865c40512cf8c72fa8f5f"
server_key_iv: "c81866947aac8dab758bd7c26b6f25db"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:07.915043 28936 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260812 03:03:07.935545 28977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:07.936854 28936 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260812 03:03:07.937196 28936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "057bcdd9d9134ac0b1b11db70b8f9c98"
format_stamp: "Formatted at 2026-08-12 03:03:07 on dist-test-slave-zpvx"
server_key: "97763d4a855865c40512cf8c72fa8f5f"
server_key_iv: "c81866947aac8dab758bd7c26b6f25db"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:07.937547 28936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:03:07.979576 28936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:03:07.984862 28936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:03:07.985339 28936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:03:08.056238 28936 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:37301
I20260812 03:03:08.056416 29067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:37301 every 8 connection(s)
I20260812 03:03:08.059095 28936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:03:08.064430 29069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:03:08.069695 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 28936
I20260812 03:03:08.070324 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:03:08.072099 10713 external_mini_cluster.cc:1428] Setting key bd5c1760af724fee2f38e5a658d0a575
I20260812 03:03:08.087505 29069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98: Bootstrap starting.
I20260812 03:03:08.092612 29069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98: Neither blocks nor log segments found. Creating new log.
Aug 12 03:03:08 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:03:08.094617 29069 log.cc:824] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98: Log is configured to *not* fsync() on all Append() calls
I20260812 03:03:08.100301 29069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98: No bootstrap required, opened a new log
I20260812 03:03:08.106837 29074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:08.075326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58400 (local address 127.10.118.126:37301)
0812 03:03:08.077801 (+ 2475us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:08.077828 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:08.077879 (+ 51us) server_negotiation.cc:407] Connection header received
0812 03:03:08.081753 (+ 3874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:08.081794 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:08.082118 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:08.082496 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:03:08.085128 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:08.086735 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:08.088732 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:08.089588 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:08.089922 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:08.095829 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:03:08.095853 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:03:08.095885 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:03:08.095982 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:03:08.100232 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:08.101360 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:08.101383 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:08.101399 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:08.101626 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:08.102387 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:08.102408 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:08.102424 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:08.103104 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:03:08.103567 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:08.104285 (+ 718us) server_negotiation.cc:299] Negotiation successful
0812 03:03:08.104924 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2036,"thread_start_us":1538,"threads_started":1}
I20260812 03:03:08.121004 29069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 37301 } }
I20260812 03:03:08.121753 29069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:03:08.122012 29069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057bcdd9d9134ac0b1b11db70b8f9c98, State: Initialized, Role: FOLLOWER
I20260812 03:03:08.122776 29069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 37301 } }
I20260812 03:03:08.123323 29069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:03:08.123605 29069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:03:08.123929 29069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:03:08.128412 29069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 37301 } }
I20260812 03:03:08.129159 29069 leader_election.cc:302] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 057bcdd9d9134ac0b1b11db70b8f9c98; no voters:
I20260812 03:03:08.130861 29069 leader_election.cc:288] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:03:08.131271 29079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:03:08.133921 29079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [term 1 LEADER]: Becoming Leader. State: Replica: 057bcdd9d9134ac0b1b11db70b8f9c98, State: Running, Role: LEADER
I20260812 03:03:08.134747 29079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 37301 } }
I20260812 03:03:08.135740 29069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:03:08.146641 29081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 37301 } } }
I20260812 03:03:08.148131 29081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [sys.catalog]: This master's current role is: LEADER
I20260812 03:03:08.147975 29082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 057bcdd9d9134ac0b1b11db70b8f9c98. Latest consensus state: current_term: 1 leader_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "057bcdd9d9134ac0b1b11db70b8f9c98" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 37301 } } }
I20260812 03:03:08.148756 29082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98 [sys.catalog]: This master's current role is: LEADER
I20260812 03:03:08.152869 29098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:03:08.163367 29098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:03:08.170794 29069 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 03:03:08.199124 29098 catalog_manager.cc:1347] Generated new cluster ID: e10daed3eefb4cc0a5a45ac65b55fc02
I20260812 03:03:08.199536 29098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:03:08.229900 29098 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:03:08.232862 29098 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:03:08.258045 29098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98: Generated new TSK 0
I20260812 03:03:08.259434 29098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:03:09.017782 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:52585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:37301
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 03:03:09 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786503789, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 03:03:09.550765 29149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:03:09.551589 29149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:03:09.552795 29149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:03:09.610378 29149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:03:09.610960 29149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:03:09.612612 29149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:03:09.676496 29149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.118.84:52585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:37301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:03:09.678229 29149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:03:09.679672 29149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:03:09.730247 29167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 12 03:03:09 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503789, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
Aug 12 03:03:10 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503789, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
W20260812 03:03:09.732650 29166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:03:09.730796 29149 server_base.cc:1034] running on GCE node
W20260812 03:03:09.737130 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:03:11.137426 29149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:03:11.140324 29149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:03:11.141793 29149 hybrid_clock.cc:648] HybridClock initialized: now 1786503791141737 us; error 83 us; skew 500 ppm
I20260812 03:03:11.142562 29149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:03:11 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503791, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:03:11.148723 29149 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:03:11.154131 29149 webserver.cc:460] Webserver started at http://127.10.118.65:45189/ using document root <none> and password file <none>
I20260812 03:03:11.155076 29149 fs_manager.cc:359] Metadata directory not provided
I20260812 03:03:11.155311 29149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:03:11.155784 29149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:03:11 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503791, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:03:11.179627 29149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "98a984f18e4b480195b91169fc95742a"
format_stamp: "Formatted at 2026-08-12 03:03:11 on dist-test-slave-zpvx"
server_key: "fc6b3b85211425c3c1c3533d98e417f5"
server_key_iv: "5d052b6b93493f446fd70d31a8bc0d19"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:11.181330 29149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "98a984f18e4b480195b91169fc95742a"
format_stamp: "Formatted at 2026-08-12 03:03:11 on dist-test-slave-zpvx"
server_key: "fc6b3b85211425c3c1c3533d98e417f5"
server_key_iv: "5d052b6b93493f446fd70d31a8bc0d19"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:11.191028 29149 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260812 03:03:11.212164 29198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:11.213451 29149 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 03:03:11.213795 29149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "98a984f18e4b480195b91169fc95742a"
format_stamp: "Formatted at 2026-08-12 03:03:11 on dist-test-slave-zpvx"
server_key: "fc6b3b85211425c3c1c3533d98e417f5"
server_key_iv: "5d052b6b93493f446fd70d31a8bc0d19"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:11.214182 29149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:03:11.273294 29149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:03:11.278798 29149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:03:11.279285 29149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:03:11.281972 29149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:03:11.286154 29149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:03:11.286358 29149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:11.286667 29149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:03:11.286836 29149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:11.426184 29149 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:34965
I20260812 03:03:11.426301 29394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:34965 every 8 connection(s)
I20260812 03:03:11.428791 29149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:03:11.430441 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 29149
I20260812 03:03:11.430961 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:03:11.432354 10713 external_mini_cluster.cc:1428] Setting key d64111af0b3e0fe9ebe97917b2ce3ddf
Aug 12 03:03:11 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503791, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:03:11.473791 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:11.434574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:44563 (local address 127.10.118.126:37301)
0812 03:03:11.436263 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:11.436306 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:11.438047 (+ 1741us) server_negotiation.cc:407] Connection header received
0812 03:03:11.445478 (+ 7431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:11.445513 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:11.445737 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:11.445929 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:03:11.449476 (+ 3547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:11.450553 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:11.454479 (+ 3926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:11.455198 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:11.455405 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:11.461734 (+ 6329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:03:11.461785 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:03:11.461822 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:03:11.461992 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:03:11.466895 (+ 4903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:11.468083 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:11.468104 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:11.468120 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:11.468323 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:11.469124 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:11.469147 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:11.469161 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:11.469674 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:03:11.469929 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:11.470978 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0812 03:03:11.471448 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":774,"threads_started":1}
I20260812 03:03:11.474439 29404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:11.434736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:37301 (local address 127.10.118.65:44563)
0812 03:03:11.437830 (+ 3094us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:11.437897 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:11.444960 (+ 7063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:11.446218 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:11.446255 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:11.447306 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:03:11.448455 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:11.448515 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:11.450825 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:11.450863 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:11.453745 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:11.453815 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:11.455549 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:11.455593 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:11.456213 (+ 620us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:11.457455 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:03:11.457572 (+ 117us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:03:11.461349 (+ 3777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:03:11.467085 (+ 5736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:03:11.467112 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:03:11.467137 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:03:11.467885 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:03:11.468545 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:03:11.468567 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:03:11.468580 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:03:11.468892 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:03:11.469902 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:03:11.469944 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:03:11.470385 (+ 441us) client_negotiation.cc:769] Sending connection context
0812 03:03:11.470737 (+ 352us) client_negotiation.cc:240] Negotiation successful
0812 03:03:11.471431 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2665,"thread_start_us":1672,"threads_started":1}
I20260812 03:03:11.476744 29395 heartbeater.cc:347] Connected to a master server at 127.10.118.126:37301
I20260812 03:03:11.477157 29395 heartbeater.cc:464] Registering TS with master...
I20260812 03:03:11.478051 29395 heartbeater.cc:511] Master 127.10.118.126:37301 requested a full tablet report, sending...
I20260812 03:03:11.480686 29008 ts_manager.cc:194] Registered new tserver with Master: 98a984f18e4b480195b91169fc95742a (127.10.118.65:34965)
I20260812 03:03:11.482740 29008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:44563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:03:11.497491 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:52585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:37301
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:03:11.809626 29408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:03:11.810102 29408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:03:11.810667 29408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:03:11.841631 29408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:03:11.841969 29408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:03:11.842850 29408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:03:11.878564 29408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.118.84:52585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:37301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:03:11.879760 29408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:03:11.880831 29408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:03:11.901762 29422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:03:12.486721 29395 heartbeater.cc:503] Master 127.10.118.126:37301 was elected leader, sending a full tablet report...
W20260812 03:03:11.901811 29421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:03:11.903122 29408 server_base.cc:1034] running on GCE node
W20260812 03:03:11.901777 29424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:03:13.101408 29408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:03:13.104123 29408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:03:13.105495 29408 hybrid_clock.cc:648] HybridClock initialized: now 1786503793105486 us; error 73 us; skew 500 ppm
I20260812 03:03:13.106302 29408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:03:13 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503793, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:03:13.116791 29408 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:03:13.121564 29408 webserver.cc:460] Webserver started at http://127.10.118.66:39723/ using document root <none> and password file <none>
I20260812 03:03:13.122514 29408 fs_manager.cc:359] Metadata directory not provided
I20260812 03:03:13.122705 29408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:03:13.123302 29408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:03:13 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503793, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:03:13.147372 29408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "4bdc92849d36470cbe4a370d9ed688c3"
format_stamp: "Formatted at 2026-08-12 03:03:13 on dist-test-slave-zpvx"
server_key: "4156c9ac1db30a939e66a951d3c83955"
server_key_iv: "568f83c056b1acac645eb27eb8838df4"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:13.149159 29408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "4bdc92849d36470cbe4a370d9ed688c3"
format_stamp: "Formatted at 2026-08-12 03:03:13 on dist-test-slave-zpvx"
server_key: "4156c9ac1db30a939e66a951d3c83955"
server_key_iv: "568f83c056b1acac645eb27eb8838df4"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:13.159672 29408 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260812 03:03:13.184352 29438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:13.185702 29408 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s
I20260812 03:03:13.186092 29408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "4bdc92849d36470cbe4a370d9ed688c3"
format_stamp: "Formatted at 2026-08-12 03:03:13 on dist-test-slave-zpvx"
server_key: "4156c9ac1db30a939e66a951d3c83955"
server_key_iv: "568f83c056b1acac645eb27eb8838df4"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:13.186569 29408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:03:13.222680 29408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:03:13.227888 29408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:03:13.228350 29408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:03:13.231014 29408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:03:13.235498 29408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:03:13.235718 29408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:13.235965 29408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:03:13.236132 29408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:13.404264 29408 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:38949
I20260812 03:03:13.404469 29642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:38949 every 8 connection(s)
I20260812 03:03:13.407004 29408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:03:13.414909 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 29408
I20260812 03:03:13.416218 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:03:13.418500 10713 external_mini_cluster.cc:1428] Setting key 6b7ce386379920b9b44c837bf9e2137f
Aug 12 03:03:13 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503793, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:03:13.453794 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:13.412132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:46521 (local address 127.10.118.126:37301)
0812 03:03:13.413600 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:13.413634 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:13.417077 (+ 3443us) server_negotiation.cc:407] Connection header received
0812 03:03:13.424534 (+ 7457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:13.424554 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:13.424717 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:13.424856 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:03:13.428259 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:13.429333 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:13.433086 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:13.433804 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:13.434051 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:13.440327 (+ 6276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:03:13.440397 (+ 70us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:03:13.440433 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:03:13.440603 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:03:13.446256 (+ 5653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:13.447547 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:13.447585 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:13.447623 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:13.447951 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:13.448836 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:13.448875 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:13.448903 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:13.449682 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:03:13.449933 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:13.451059 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0812 03:03:13.451421 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":536,"threads_started":1}
I20260812 03:03:13.454173 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:13.412411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:37301 (local address 127.10.118.66:46521)
0812 03:03:13.416640 (+ 4229us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:13.416848 (+ 208us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:13.424164 (+ 7316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:13.425179 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:13.425216 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:13.426477 (+ 1261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:03:13.427923 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:13.427997 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:13.429650 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:13.429709 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:13.432754 (+ 3045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:13.432825 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:13.434147 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:13.434191 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:13.434918 (+ 727us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:13.436043 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:03:13.436303 (+ 260us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:03:13.439885 (+ 3582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:03:13.446509 (+ 6624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:03:13.446540 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:03:13.446565 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:03:13.447280 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:03:13.448197 (+ 917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:03:13.448221 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:03:13.448235 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:03:13.448592 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:03:13.449920 (+ 1328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:03:13.449951 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:03:13.450497 (+ 546us) client_negotiation.cc:769] Sending connection context
0812 03:03:13.450826 (+ 329us) client_negotiation.cc:240] Negotiation successful
0812 03:03:13.451342 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3526,"thread_start_us":2173,"threads_started":1}
I20260812 03:03:13.456574 29643 heartbeater.cc:347] Connected to a master server at 127.10.118.126:37301
I20260812 03:03:13.456991 29643 heartbeater.cc:464] Registering TS with master...
I20260812 03:03:13.458129 29643 heartbeater.cc:511] Master 127.10.118.126:37301 requested a full tablet report, sending...
I20260812 03:03:13.460578 29008 ts_manager.cc:194] Registered new tserver with Master: 4bdc92849d36470cbe4a370d9ed688c3 (127.10.118.66:38949)
I20260812 03:03:13.463004 29008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:46521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:03:13.483122 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:52585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:37301
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:03:13.718421 29386 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0840 after lost signal to thread 29158
W20260812 03:03:13.719259 29386 debug-util.cc:398] Leaking SignalData structure 0x7b0800041340 after lost signal to thread 29394
W20260812 03:03:13.844785 29652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:03:13.845388 29652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:03:13.846113 29652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:03:13.880020 29652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:03:13.880465 29652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:03:13.881417 29652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:03:13.918136 29652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.118.84:52585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:37301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:03:13.919278 29652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:03:13.920293 29652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:03:13.946194 29663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:03:14.466840 29643 heartbeater.cc:503] Master 127.10.118.126:37301 was elected leader, sending a full tablet report...
W20260812 03:03:13.948169 29666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:03:13.946688 29662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:03:15.249575 29665 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1300 milliseconds
I20260812 03:03:15.249706 29652 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:03:15.251035 29652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:03:15.253857 29652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:03:15.255280 29652 hybrid_clock.cc:648] HybridClock initialized: now 1786503795255229 us; error 71 us; skew 500 ppm
I20260812 03:03:15.256239 29652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:03:15 dist-test-slave-zpvx krb5kdc[28161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503795, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:03:15.262760 29652 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:03:15.268941 29652 webserver.cc:460] Webserver started at http://127.10.118.67:35349/ using document root <none> and password file <none>
I20260812 03:03:15.269891 29652 fs_manager.cc:359] Metadata directory not provided
I20260812 03:03:15.270118 29652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:03:15.270578 29652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:03:15 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503795, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:03:15.296135 29652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "1bbcaed85ceb4fc794664ee30dd635cd"
format_stamp: "Formatted at 2026-08-12 03:03:15 on dist-test-slave-zpvx"
server_key: "73fbb409585a3ffd846a6b17ed1e10bf"
server_key_iv: "ae3b40fd0083e464431f100c5e4e9d7e"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:15.297874 29652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "1bbcaed85ceb4fc794664ee30dd635cd"
format_stamp: "Formatted at 2026-08-12 03:03:15 on dist-test-slave-zpvx"
server_key: "73fbb409585a3ffd846a6b17ed1e10bf"
server_key_iv: "ae3b40fd0083e464431f100c5e4e9d7e"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:15.307879 29652 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20260812 03:03:15.334291 29682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:15.335768 29652 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260812 03:03:15.336076 29652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "1bbcaed85ceb4fc794664ee30dd635cd"
format_stamp: "Formatted at 2026-08-12 03:03:15 on dist-test-slave-zpvx"
server_key: "73fbb409585a3ffd846a6b17ed1e10bf"
server_key_iv: "ae3b40fd0083e464431f100c5e4e9d7e"
server_key_version: "kuduclusterkey@0"
I20260812 03:03:15.336580 29652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:03:15.371544 29652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:03:15.377040 29652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:03:15.377486 29652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:03:15.380316 29652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:03:15.384604 29652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:03:15.384836 29652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:15.385093 29652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:03:15.385264 29652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:03:15.587088 29652 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:42603
I20260812 03:03:15.587222 29858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:42603 every 8 connection(s)
I20260812 03:03:15.589906 29652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:03:15.592754 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 29652
I20260812 03:03:15.593327 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:03:15.595193 10713 external_mini_cluster.cc:1428] Setting key 59d19e23727015d7ae40413dc7343a95
Aug 12 03:03:15 dist-test-slave-zpvx krb5kdc[28161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503795, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:03:15.647431 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.597187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:37861 (local address 127.10.118.126:37301)
0812 03:03:15.608473 (+ 11286us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:15.608510 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:15.608564 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:03:15.616483 (+ 7919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:15.616539 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:15.616843 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:15.617191 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:03:15.620620 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.622230 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.625226 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.626175 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.626486 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.633974 (+ 7488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:03:15.634027 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:03:15.634054 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:03:15.634277 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:03:15.638671 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:15.640272 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:15.640298 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:15.640313 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:15.640667 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:15.641980 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:15.642017 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:15.642041 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:15.642677 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:03:15.642976 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:15.644592 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0812 03:03:15.645259 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10649,"thread_start_us":10241,"threads_started":1}
I20260812 03:03:15.649530 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.597873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:37301 (local address 127.10.118.67:37861)
0812 03:03:15.605993 (+ 8120us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:15.606191 (+ 198us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:15.616052 (+ 9861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:15.617408 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:15.617449 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:15.618563 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:03:15.620274 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.620321 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.622481 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.622533 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.624963 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.625007 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.626554 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.626591 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.627311 (+ 720us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.628670 (+ 1359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:03:15.628778 (+ 108us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:03:15.633516 (+ 4738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:03:15.638925 (+ 5409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:03:15.638967 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:03:15.638995 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:03:15.640025 (+ 1030us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:03:15.640991 (+ 966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:03:15.641033 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:03:15.641061 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:03:15.641495 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:03:15.642965 (+ 1470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:03:15.643018 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:03:15.643741 (+ 723us) client_negotiation.cc:769] Sending connection context
0812 03:03:15.644255 (+ 514us) client_negotiation.cc:240] Negotiation successful
0812 03:03:15.645157 (+ 902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7383,"thread_start_us":6872,"threads_started":1}
I20260812 03:03:15.651497 29859 heartbeater.cc:347] Connected to a master server at 127.10.118.126:37301
I20260812 03:03:15.651911 29859 heartbeater.cc:464] Registering TS with master...
I20260812 03:03:15.653237 29859 heartbeater.cc:511] Master 127.10.118.126:37301 requested a full tablet report, sending...
I20260812 03:03:15.655997 29008 ts_manager.cc:194] Registered new tserver with Master: 1bbcaed85ceb4fc794664ee30dd635cd (127.10.118.67:42603)
I20260812 03:03:15.658679 29008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:37861
I20260812 03:03:15.661267 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:03:15.720232 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.685517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.10.118.126:37301)
0812 03:03:15.686244 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:15.686282 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:15.687465 (+ 1183us) server_negotiation.cc:407] Connection header received
0812 03:03:15.688004 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:15.688022 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:15.688236 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:15.688422 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:03:15.694257 (+ 5835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.695299 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.698993 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.699783 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.700053 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.706054 (+ 6001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:03:15.706075 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:03:15.706092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:03:15.706231 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:03:15.710390 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:15.712002 (+ 1612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:15.712024 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:15.712079 (+ 55us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:15.712339 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:03:15.713244 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:03:15.713285 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:03:15.713344 (+ 59us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:03:15.714304 (+ 960us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:03:15.714671 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:15.715985 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0812 03:03:15.717391 (+ 1406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260812 03:03:15.727819 29007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:03:15.756357 29007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:03:15.850343 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.803109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:34965 (local address 127.0.0.1:49768)
0812 03:03:15.823097 (+ 19988us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:15.823386 (+ 289us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:15.827716 (+ 4330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:15.829682 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:15.829716 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:15.829864 (+ 148us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:03:15.831068 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.831159 (+ 91us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.835087 (+ 3928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.835121 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.839781 (+ 4660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.839867 (+ 86us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.845726 (+ 5859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.845764 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.846305 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.846471 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 03:03:15.846628 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 03:03:15.847033 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19491,"thread_start_us":5352,"threads_started":1}
I20260812 03:03:15.854751 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.800776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.10.118.66:38949)
0812 03:03:15.812556 (+ 11780us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:15.812594 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:15.812657 (+ 63us) server_negotiation.cc:407] Connection header received
0812 03:03:15.812912 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:15.812946 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:15.813373 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:15.813629 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:03:15.824431 (+ 10802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.828124 (+ 3693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.838485 (+ 10361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.841004 (+ 2519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.841726 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.842749 (+ 1023us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:15.851924 (+ 9175us) server_negotiation.cc:299] Negotiation successful
0812 03:03:15.852324 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11192,"thread_start_us":13422,"threads_started":1}
I20260812 03:03:15.854987 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.806498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.10.118.65:34965)
0812 03:03:15.821273 (+ 14775us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:15.821307 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:15.828036 (+ 6729us) server_negotiation.cc:407] Connection header received
0812 03:03:15.828277 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:15.828316 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:15.828751 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:15.829035 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:03:15.831530 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.834739 (+ 3209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.840222 (+ 5483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.845293 (+ 5071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.848451 (+ 3158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.849029 (+ 578us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:15.849199 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 03:03:15.852475 (+ 3276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14063,"thread_start_us":14576,"threads_started":1}
I20260812 03:03:15.848147 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.800660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:38949 (local address 127.0.0.1:33124)
0812 03:03:15.809201 (+ 8541us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:15.809333 (+ 132us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:15.809892 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:15.813724 (+ 3832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:15.813756 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:15.813874 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:03:15.823875 (+ 10001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.823957 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.828531 (+ 4574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.828581 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.835190 (+ 6609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.835342 (+ 152us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.843184 (+ 7842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.843271 (+ 87us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.844194 (+ 923us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.844470 (+ 276us) client_negotiation.cc:769] Sending connection context
0812 03:03:15.844766 (+ 296us) client_negotiation.cc:240] Negotiation successful
0812 03:03:15.845335 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7788,"thread_start_us":22416,"threads_started":1}
I20260812 03:03:15.861260 29294 tablet_service.cc:1467] Processing CreateTablet for tablet d6817e4256b340d39a00b049c3b43f1e (DEFAULT_TABLE table=test-table [id=61a2d362fa094544ab9d43b33ca712ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:03:15.862893 29524 tablet_service.cc:1467] Processing CreateTablet for tablet d6817e4256b340d39a00b049c3b43f1e (DEFAULT_TABLE table=test-table [id=61a2d362fa094544ab9d43b33ca712ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:03:15.864022 29294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6817e4256b340d39a00b049c3b43f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:03:15.865998 29524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6817e4256b340d39a00b049c3b43f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:03:15.880897 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.816987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42603 (local address 127.0.0.1:53484)
0812 03:03:15.824574 (+ 7587us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:15.824660 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:15.824933 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:15.854200 (+ 29267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:15.854256 (+ 56us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:15.854400 (+ 144us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:03:15.855498 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.855565 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.864610 (+ 9045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.864677 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.868556 (+ 3879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:15.868632 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.876331 (+ 7699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.876390 (+ 59us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:15.876924 (+ 534us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.877094 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 03:03:15.877238 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 03:03:15.877598 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6968,"thread_start_us":5082,"threads_started":1}
I20260812 03:03:15.888340 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:15.816637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.10.118.67:42603)
0812 03:03:15.852692 (+ 36055us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:15.852725 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:15.852785 (+ 60us) server_negotiation.cc:407] Connection header received
0812 03:03:15.852985 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:15.853022 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:15.853434 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:15.855835 (+ 2401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:03:15.856905 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.864162 (+ 7257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.869046 (+ 4884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:15.875802 (+ 6756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:15.882162 (+ 6360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:15.885531 (+ 3369us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:15.885692 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 03:03:15.886030 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35501,"thread_start_us":34679,"threads_started":1}
I20260812 03:03:15.894527 29757 tablet_service.cc:1467] Processing CreateTablet for tablet d6817e4256b340d39a00b049c3b43f1e (DEFAULT_TABLE table=test-table [id=61a2d362fa094544ab9d43b33ca712ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:03:15.900508 29757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6817e4256b340d39a00b049c3b43f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:03:15.910043 29890 tablet_bootstrap.cc:492] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Bootstrap starting.
I20260812 03:03:15.916136 29892 tablet_bootstrap.cc:492] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a: Bootstrap starting.
I20260812 03:03:15.928462 29890 tablet_bootstrap.cc:654] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Neither blocks nor log segments found. Creating new log.
I20260812 03:03:15.930573 29892 tablet_bootstrap.cc:654] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a: Neither blocks nor log segments found. Creating new log.
I20260812 03:03:15.932502 29890 log.cc:824] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Log is configured to *not* fsync() on all Append() calls
I20260812 03:03:15.935853 29892 log.cc:824] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a: Log is configured to *not* fsync() on all Append() calls
I20260812 03:03:15.950089 29899 tablet_bootstrap.cc:492] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Bootstrap starting.
I20260812 03:03:15.954530 29890 tablet_bootstrap.cc:492] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: No bootstrap required, opened a new log
I20260812 03:03:15.955330 29890 ts_tablet_manager.cc:1397] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.013s
I20260812 03:03:15.955480 29892 tablet_bootstrap.cc:492] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a: No bootstrap required, opened a new log
I20260812 03:03:15.956141 29892 ts_tablet_manager.cc:1397] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.021s
I20260812 03:03:15.961781 29899 tablet_bootstrap.cc:654] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Neither blocks nor log segments found. Creating new log.
I20260812 03:03:15.966063 29899 log.cc:824] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Log is configured to *not* fsync() on all Append() calls
I20260812 03:03:15.981304 29899 tablet_bootstrap.cc:492] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: No bootstrap required, opened a new log
I20260812 03:03:15.981941 29899 ts_tablet_manager.cc:1397] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.015s
I20260812 03:03:15.998324 29890 raft_consensus.cc:348] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:15.999531 29890 raft_consensus.cc:374] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:03:15.999943 29890 raft_consensus.cc:729] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdc92849d36470cbe4a370d9ed688c3, State: Initialized, Role: FOLLOWER
I20260812 03:03:16.001231 29890 consensus_queue.cc:260] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.006904 29892 raft_consensus.cc:348] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.008805 29892 raft_consensus.cc:374] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:03:16.009435 29892 raft_consensus.cc:729] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a984f18e4b480195b91169fc95742a, State: Initialized, Role: FOLLOWER
I20260812 03:03:16.010823 29892 consensus_queue.cc:260] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.016865 29890 ts_tablet_manager.cc:1428] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Time spent starting tablet: real 0.061s user 0.039s sys 0.020s
I20260812 03:03:16.034663 29892 ts_tablet_manager.cc:1428] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a: Time spent starting tablet: real 0.078s user 0.053s sys 0.017s
I20260812 03:03:16.035948 29903 raft_consensus.cc:482] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:03:16.030814 29899 raft_consensus.cc:348] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.036806 29899 raft_consensus.cc:374] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:03:16.037142 29899 raft_consensus.cc:729] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbcaed85ceb4fc794664ee30dd635cd, State: Initialized, Role: FOLLOWER
I20260812 03:03:16.036670 29903 raft_consensus.cc:504] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.038281 29899 consensus_queue.cc:260] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.051859 29903 leader_election.cc:288] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bdc92849d36470cbe4a370d9ed688c3 (127.10.118.66:38949), 1bbcaed85ceb4fc794664ee30dd635cd (127.10.118.67:42603)
I20260812 03:03:16.053892 29859 heartbeater.cc:503] Master 127.10.118.126:37301 was elected leader, sending a full tablet report...
I20260812 03:03:16.055315 29899 ts_tablet_manager.cc:1428] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Time spent starting tablet: real 0.073s user 0.039s sys 0.015s
I20260812 03:03:16.072253 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.041852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:35899 (local address 127.10.118.66:38949)
0812 03:03:16.042492 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:16.042523 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:16.048652 (+ 6129us) server_negotiation.cc:407] Connection header received
0812 03:03:16.050966 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:16.050996 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:16.051308 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:16.051453 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:03:16.058367 (+ 6914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.060587 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.065350 (+ 4763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.067404 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.067896 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:16.068362 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:16.070339 (+ 1977us) server_negotiation.cc:299] Negotiation successful
0812 03:03:16.070694 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260812 03:03:16.077416 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.041626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:38949 (local address 127.10.118.65:35899)
0812 03:03:16.048434 (+ 6808us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:16.048498 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:16.050121 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:16.056090 (+ 5969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:16.056207 (+ 117us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:16.056316 (+ 109us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:03:16.057874 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:16.057942 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.061410 (+ 3468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.061452 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:16.064983 (+ 3531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:16.065044 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.068970 (+ 3926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.069012 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:16.069691 (+ 679us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:16.070015 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 03:03:16.072936 (+ 2921us) client_negotiation.cc:240] Negotiation successful
0812 03:03:16.073945 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6182,"mutex_wait_us":506,"thread_start_us":4672,"threads_started":1}
I20260812 03:03:16.079857 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6817e4256b340d39a00b049c3b43f1e" candidate_uuid: "98a984f18e4b480195b91169fc95742a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdc92849d36470cbe4a370d9ed688c3" is_pre_election: true
I20260812 03:03:16.081053 29556 raft_consensus.cc:2454] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a984f18e4b480195b91169fc95742a in term 0.
I20260812 03:03:16.083078 29204 leader_election.cc:302] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdc92849d36470cbe4a370d9ed688c3, 98a984f18e4b480195b91169fc95742a; no voters:
I20260812 03:03:16.084479 29903 raft_consensus.cc:2781] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:03:16.084833 29903 raft_consensus.cc:482] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:03:16.085263 29903 raft_consensus.cc:3037] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:03:16.088809 29887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.052449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:41779 (local address 127.10.118.67:42603)
0812 03:03:16.053134 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:16.053178 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:16.071203 (+ 18025us) server_negotiation.cc:407] Connection header received
0812 03:03:16.071712 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:16.071740 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:16.071976 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:16.072113 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:03:16.074626 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.076010 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.081859 (+ 5849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.083758 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.084192 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:16.084492 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:16.086238 (+ 1746us) server_negotiation.cc:299] Negotiation successful
0812 03:03:16.086620 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260812 03:03:16.092981 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.050417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42603 (local address 127.10.118.65:41779)
0812 03:03:16.070993 (+ 20576us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:16.071053 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 03:03:16.071442 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:03:16.072639 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:03:16.072670 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:03:16.072766 (+ 96us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:03:16.074272 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:16.074324 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.076528 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.076570 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:16.081389 (+ 4819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:03:16.081456 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.085019 (+ 3563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.085058 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:03:16.085608 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:16.085876 (+ 268us) client_negotiation.cc:769] Sending connection context
0812 03:03:16.089655 (+ 3779us) client_negotiation.cc:240] Negotiation successful
0812 03:03:16.090039 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20109,"thread_start_us":22712,"threads_started":1}
I20260812 03:03:16.095431 29784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6817e4256b340d39a00b049c3b43f1e" candidate_uuid: "98a984f18e4b480195b91169fc95742a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" is_pre_election: true
I20260812 03:03:16.096679 29784 raft_consensus.cc:2454] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a984f18e4b480195b91169fc95742a in term 0.
I20260812 03:03:16.099610 29903 raft_consensus.cc:504] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.102955 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6817e4256b340d39a00b049c3b43f1e" candidate_uuid: "98a984f18e4b480195b91169fc95742a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bdc92849d36470cbe4a370d9ed688c3"
I20260812 03:03:16.103333 29784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6817e4256b340d39a00b049c3b43f1e" candidate_uuid: "98a984f18e4b480195b91169fc95742a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbcaed85ceb4fc794664ee30dd635cd"
I20260812 03:03:16.103761 29556 raft_consensus.cc:3037] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:03:16.103909 29784 raft_consensus.cc:3037] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 0 FOLLOWER]: Advancing to term 1
W20260812 03:03:16.111539 29860 tablet.cc:2369] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:03:16.113991 29556 raft_consensus.cc:2454] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a984f18e4b480195b91169fc95742a in term 1.
I20260812 03:03:16.113989 29784 raft_consensus.cc:2454] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a984f18e4b480195b91169fc95742a in term 1.
I20260812 03:03:16.115285 29204 leader_election.cc:302] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdc92849d36470cbe4a370d9ed688c3, 98a984f18e4b480195b91169fc95742a; no voters:
I20260812 03:03:16.115475 29903 leader_election.cc:288] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [CANDIDATE]: Term 1 election: Requested vote from peers 4bdc92849d36470cbe4a370d9ed688c3 (127.10.118.66:38949), 1bbcaed85ceb4fc794664ee30dd635cd (127.10.118.67:42603)
I20260812 03:03:16.117005 29903 raft_consensus.cc:2781] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:03:16.120730 29903 raft_consensus.cc:686] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [term 1 LEADER]: Becoming Leader. State: Replica: 98a984f18e4b480195b91169fc95742a, State: Running, Role: LEADER
I20260812 03:03:16.122442 29903 consensus_queue.cc:237] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } }
I20260812 03:03:16.137171 29006 catalog_manager.cc:5495] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a984f18e4b480195b91169fc95742a (127.10.118.65). New cstate: current_term: 1 leader_uuid: "98a984f18e4b480195b91169fc95742a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a984f18e4b480195b91169fc95742a" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 } health_report { overall_health: UNKNOWN } } }
W20260812 03:03:16.164763 29645 tablet.cc:2369] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:03:16.182314 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.171331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.10.118.65:34965)
0812 03:03:16.171740 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0812 03:03:16.171759 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:03:16.171828 (+ 69us) server_negotiation.cc:407] Connection header received
0812 03:03:16.172340 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:03:16.172358 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:03:16.172642 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:03:16.172836 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:03:16.173808 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.174790 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.178008 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:03:16.178773 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:03:16.179063 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:03:16.180009 (+ 946us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:03:16.180561 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:03:16.180704 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 03:03:16.180994 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 03:03:16.181130 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260812 03:03:16.189822 29399 tablet.cc:2369] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:03:16.201253 29784 raft_consensus.cc:1261] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 1 FOLLOWER]: Refusing update from remote peer 98a984f18e4b480195b91169fc95742a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:03:16.201997 29556 raft_consensus.cc:1261] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 1 FOLLOWER]: Refusing update from remote peer 98a984f18e4b480195b91169fc95742a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:03:16.203415 29907 consensus_queue.cc:1035] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bbcaed85ceb4fc794664ee30dd635cd" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 03:03:16.204304 29903 consensus_queue.cc:1035] T d6817e4256b340d39a00b049c3b43f1e P 98a984f18e4b480195b91169fc95742a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bdc92849d36470cbe4a370d9ed688c3" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 38949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:03:16.230265 29911 mvcc.cc:205] Tried to move back new op lower bound from 7317519549219262464 to 7317519548927913984. Current Snapshot: MvccSnapshot[applied={T|T < 7317519549219262464}]
I20260812 03:03:16.277187 29006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:03:16.277853 29006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:03:16.288446 29006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 057bcdd9d9134ac0b1b11db70b8f9c98: Sending DeleteTablet for 3 replicas of tablet d6817e4256b340d39a00b049c3b43f1e
I20260812 03:03:16.290701 29524 tablet_service.cc:1514] Processing DeleteTablet for tablet d6817e4256b340d39a00b049c3b43f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:03:16 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:33124
I20260812 03:03:16.291355 29294 tablet_service.cc:1514] Processing DeleteTablet for tablet d6817e4256b340d39a00b049c3b43f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:03:16 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:49768
I20260812 03:03:16.293946 29757 tablet_service.cc:1514] Processing DeleteTablet for tablet d6817e4256b340d39a00b049c3b43f1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:03:16 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:53484
I20260812 03:03:16.294592 29932 tablet_replica.cc:330] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: stopping tablet replica
I20260812 03:03:16.295009 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 29149
I20260812 03:03:16.296058 29932 raft_consensus.cc:2229] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:03:16.297117 29932 raft_consensus.cc:2258] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:03:16.298815 29933 tablet_replica.cc:330] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: stopping tablet replica
I20260812 03:03:16.299657 29933 raft_consensus.cc:2229] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:03:16.300235 29933 raft_consensus.cc:2258] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:03:16.305256 29932 ts_tablet_manager.cc:1905] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:03:16.305377 29933 ts_tablet_manager.cc:1905] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:03:16.320161 29933 ts_tablet_manager.cc:1918] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:03:16.320165 29932 ts_tablet_manager.cc:1918] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:03:16.320580 29932 log.cc:1196] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/d6817e4256b340d39a00b049c3b43f1e
I20260812 03:03:16.320587 29933 log.cc:1196] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/d6817e4256b340d39a00b049c3b43f1e
I20260812 03:03:16.321391 29933 ts_tablet_manager.cc:1939] T d6817e4256b340d39a00b049c3b43f1e P 1bbcaed85ceb4fc794664ee30dd635cd: Deleting consensus metadata
I20260812 03:03:16.321389 29932 ts_tablet_manager.cc:1939] T d6817e4256b340d39a00b049c3b43f1e P 4bdc92849d36470cbe4a370d9ed688c3: Deleting consensus metadata
I20260812 03:03:16.324604 28980 catalog_manager.cc:4841] TS 1bbcaed85ceb4fc794664ee30dd635cd (127.10.118.67:42603): tablet d6817e4256b340d39a00b049c3b43f1e (table test-table [id=61a2d362fa094544ab9d43b33ca712ca]) successfully deleted
I20260812 03:03:16.325230 28982 catalog_manager.cc:4841] TS 4bdc92849d36470cbe4a370d9ed688c3 (127.10.118.66:38949): tablet d6817e4256b340d39a00b049c3b43f1e (table test-table [id=61a2d362fa094544ab9d43b33ca712ca]) successfully deleted
W20260812 03:03:16.336398 28988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:34965 (error 108)
I20260812 03:03:16.336675 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 29408
I20260812 03:03:16.350661 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.349622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:34965 (local address 127.0.0.1:49784)
0812 03:03:16.350083 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:16.350256 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:34965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260812 03:03:16.351076 28988 catalog_manager.cc:4584] TS 98a984f18e4b480195b91169fc95742a (127.10.118.65:34965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6817e4256b340d39a00b049c3b43f1e: Network error: Client connection negotiation failed: client connection to 127.10.118.65:34965: connect: Connection refused (error 111)
I20260812 03:03:16.376165 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 29652
I20260812 03:03:16.406805 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.405206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:34965 (local address 127.0.0.1:49800)
0812 03:03:16.405890 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:16.406141 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:34965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 03:03:16.409636 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:03:16.408188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:34965 (local address 127.0.0.1:49814)
0812 03:03:16.408799 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0812 03:03:16.409026 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:34965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 03:03:16.410877 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 28936
I20260812 03:03:16.467530 10713 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 03:03:16.815220 10713 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 03:03:16.815619 10713 mini_ranger.cc:66] Stopping Ranger...
I20260812 03:03:17.160754 10713 mini_ranger.cc:68] Stopped Ranger
2026-08-12 03:03:17.161 UTC [28201] LOG: received smart shutdown request
2026-08-12 03:03:17.165 UTC [28201] LOG: background worker "logical replication launcher" (PID 28218) exited with exit code 1
2026-08-12 03:03:17.165 UTC [28209] LOG: shutting down
2026-08-12 03:03:17.330 UTC [28201] LOG: database system is shut down
2026-08-12T03:03:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57208 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:03:17 dist-test-slave-zpvx krb5kdc[29952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:03:17 dist-test-slave-zpvx krb5kdc[29952](info): set up 2 sockets
Aug 12 03:03:17 dist-test-slave-zpvx krb5kdc[29952](info): commencing operation
krb5kdc: starting...
W20260812 03:03:18.855484 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:03:19 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:03:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:03:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/rangeradmin_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/rangeradmin_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/rangerlookup_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/rangerlookup_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
I20260812 03:03:19.189538 10713 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgres/pg_ctl -D /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/postgres -l logfile start
2026-08-12 03:03:21.692 UTC [30012] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 03:03:21.693 UTC [30012] LOG: listening on IPv4 address "127.10.118.84", port 56057
2026-08-12 03:03:21.696 UTC [30012] LOG: listening on Unix socket "/tmp/.s.PGSQL.56057"
2026-08-12 03:03:21.717 UTC [30015] LOG: database system was shut down at 2026-08-12 03:03:20 UTC
2026-08-12 03:03:21.721 UTC [30012] LOG: database system is ready to accept connections
I20260812 03:03:22.953315 10713 mini_postgres.cc:96] Postgres bound to 56057
2026-08-12 03:03:22.972 UTC [30027] FATAL: database "slave" does not exist
127.10.118.84:56057 - accepting connections
I20260812 03:03:22.973538 10713 mini_ranger.cc:161] Starting Ranger...
I20260812 03:03:23.000087 10713 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 03:03:23.476156 10713 mini_ranger.cc:87] Created ranger Postgres database
I20260812 03:03:23.476531 10713 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-admin
2026-08-12 03:03:23,768 [I] DB FLAVOR :POSTGRES
2026-08-12 03:03:23,769 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 03:03:23,769 [I] Checking connection..
2026-08-12 03:03:23,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 03:03:24,016 [I] Checking connection passed.
2026-08-12 03:03:24,016 [I] --------- Verifying version history table ---------
2026-08-12 03:03:24,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 03:03:24,263 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 03:03:24,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 03:03:24,502 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 03:03:24,503 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 03:03:24,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 03:03:24.764 UTC [30139] WARNING: there is no transaction in progress
2026-08-12 03:03:24,771 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 03:03:24,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 03:03:25,040 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 03:03:25,040 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 03:03:25,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 03:03:25,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 03:03:25,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zpvx.c.gcp-upstream.internal','N') ;"
2026-08-12 03:03:25,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 03:03:26,060 [I] Table x_portal_user does not exist in database ranger
2026-08-12 03:03:26,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 03:03:26,326 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 03:03:26,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 03:03:26,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 03:03:26,874 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 03:03:26,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 03:03:27.799 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.814 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.830 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.846 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.862 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.880 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.948 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.955 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.970 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.980 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:27.990 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:28.003 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:28.013 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:28.019 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:28.025 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:28.032 UTC [30314] WARNING: there is no transaction in progress
2026-08-12 03:03:28,669 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 03:03:28,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 03:03:28,935 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 03:03:28,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 03:03:29,172 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 03:03:29,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 03:03:29,459 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 03:03:29,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 03:03:29,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 03:03:29,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zpvx.c.gcp-upstream.internal';"
2026-08-12 03:03:30,220 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 03:03:30,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 03:03:30,457 [I] DB_PATCHES have already been applied
I20260812 03:03:30.463260 10713 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-admin:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskx4bzWQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 03:03:30.463510 10713 mini_ranger.cc:193] Using host: 127.10.118.84
I20260812 03:03:30.479647 10713 mini_ranger.cc:227] Ranger admin URL: http://127.10.118.84:56103
Aug 12, 2026 3:03:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 3:03:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 3:03:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 3:03:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 3:03:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 3:03:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 3:03:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 03:03:31.928097 10713 mini_ranger.cc:160] Time spent starting Ranger: real 8.955s user 0.008s sys 0.042s
Aug 12, 2026 3:03:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 3:03:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 3:03:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 3:03:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 3:03:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 03:03:54.312757 10713 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.118.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.118.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/rangerkms_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.118.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/rangerkms_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.118.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.118.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.118.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/HTTP_127.10.118.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 03:03:54 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503834, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 03:03:54.497978 10713 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 03:03:54.522670 10713 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 03:03:55.380340 10713 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 03:03:55.380736 10713 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms
I20260812 03:03:55.380947 10713 mini_ranger_kms.cc:227] Using postgres at 127.10.118.84:56057
2026-08-12 03:03:56,270 [I] DB FLAVOR :POSTGRES
2026-08-12 03:03:56,270 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 03:03:56,270 [I] Checking connection
2026-08-12 03:03:56,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 03:03:56,503 [I] connection success
2026-08-12 03:03:56,503 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 03:03:56,503 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 03:03:56,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 03:03:56,773 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 03:03:56,773 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 03:03:56,773 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 03:03:56,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.118.84:56057/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 03:03:57,071 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 03:03:57.579144 10713 mini_ranger_kms.cc:310] Created kms service
I20260812 03:03:57.776268 10713 mini_ranger_kms.cc:326] Created kudu user
I20260812 03:03:57.820501 10713 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 03:03:57 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503834, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:03:58.260924 10713 mini_ranger_kms.cc:384] Added ranger policy
I20260812 03:03:58.261458 10713 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms:/tmp/dist-test-taskx4bzWQ/build/tsan/bin/postgresql.jar:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskx4bzWQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 03:03:58.261655 10713 mini_ranger_kms.cc:240] Using host: 127.10.118.84
I20260812 03:03:58.278993 10713 mini_ranger_kms.cc:276] Ranger KMS PID: 30648
I20260812 03:03:58.279356 10713 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.118.84:45357
Aug 12, 2026 3:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 3:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 3:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 3:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 3:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 3:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 03:04:00 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786503840, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.118.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:01.230041 10713 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.732s user 0.346s sys 0.578s
I20260812 03:04:01.230599 10713 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 03:04:01.231048 10713 mini_ranger_kms.cc:401] 127.10.118.84:45357/kms/v1/keys
Aug 12 03:04:01 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503840, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.118.84@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
Aug 12 03:04:01 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503840, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.118.84@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
Aug 12 03:04:03 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:04:04.017422 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41899
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:41899
--ranger_config_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:04.358527 30725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:04.359181 30725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:04.359795 30725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:04.391799 30725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:04:04.392231 30725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:04.392566 30725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:04.392870 30725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:04:04.393191 30725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:04:04.430313 30725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.118.84:45357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:41899
--ranger_config_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:04.431557 30725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:04.432698 30725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:04.458567 30732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:04.458730 30734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:04.459589 30737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:05.667850 30735 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1207 milliseconds
I20260812 03:04:05.667982 30725 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:04:05.669301 30725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:05.672540 30725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:05.673969 30725 hybrid_clock.cc:648] HybridClock initialized: now 1786503845673940 us; error 70 us; skew 500 ppm
I20260812 03:04:05.674777 30725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:05 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:05.685452 30725 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:04:05.690639 30725 webserver.cc:460] Webserver started at http://127.10.118.126:40001/ using document root <none> and password file <none>
I20260812 03:04:05.691540 30725 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:05.691747 30725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:05.692214 30725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:04:05 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503845, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:05.745863 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "304aa96dc0514b9dace7b3f2ef7fcf14"
format_stamp: "Formatted at 2026-08-12 03:04:05 on dist-test-slave-zpvx"
server_key: "2fb8570bff42cc94341aefb8e6d594c5"
server_key_iv: "89f2618fd6b14a295d7dcc0b3d33d4e2"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:05.747081 30725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "304aa96dc0514b9dace7b3f2ef7fcf14"
format_stamp: "Formatted at 2026-08-12 03:04:05 on dist-test-slave-zpvx"
server_key: "2fb8570bff42cc94341aefb8e6d594c5"
server_key_iv: "89f2618fd6b14a295d7dcc0b3d33d4e2"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:05.755168 30725 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s
I20260812 03:04:05.779814 30747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:05.781116 30725 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 03:04:05.781474 30725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "304aa96dc0514b9dace7b3f2ef7fcf14"
format_stamp: "Formatted at 2026-08-12 03:04:05 on dist-test-slave-zpvx"
server_key: "2fb8570bff42cc94341aefb8e6d594c5"
server_key_iv: "89f2618fd6b14a295d7dcc0b3d33d4e2"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:05.781860 30725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:05.826296 30725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:05.831596 30725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:05.832078 30725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:05.903646 30725 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:41899
I20260812 03:04:05.903723 30836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:41899 every 8 connection(s)
I20260812 03:04:05.906407 30725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:04:05.908330 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 30725
I20260812 03:04:05.909027 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:04:05.910830 10713 external_mini_cluster.cc:1428] Setting key 05927d21d568e6be1e30c592ccffbeef
I20260812 03:04:05.912750 30839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 03:04:05 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:05.938814 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Bootstrap starting.
I20260812 03:04:05.945947 30839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:05.948136 30839 log.cc:824] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:05.952399 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:05.913457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.10.118.126:41899)
0812 03:04:05.916078 (+ 2621us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:05.916118 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:05.916191 (+ 73us) server_negotiation.cc:407] Connection header received
0812 03:04:05.923093 (+ 6902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:05.923150 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:05.923654 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:05.924129 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:05.926822 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:05.929421 (+ 2599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:05.931386 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:05.932747 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:05.933097 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:05.938626 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:05.938661 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:05.938701 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:05.938879 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:05.944646 (+ 5767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:05.945707 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:05.945756 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:05.945798 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:05.946109 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:05.946809 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:05.946845 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:05.946876 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:05.947861 (+ 985us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:05.948441 (+ 580us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:05.949187 (+ 746us) server_negotiation.cc:299] Negotiation successful
0812 03:04:05.949838 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2045,"thread_start_us":1626,"threads_started":1}
I20260812 03:04:05.954397 30839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: No bootstrap required, opened a new log
I20260812 03:04:05.975354 30839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:05.976020 30839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:05.976307 30839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304aa96dc0514b9dace7b3f2ef7fcf14, State: Initialized, Role: FOLLOWER
I20260812 03:04:05.977133 30839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:05.977741 30839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:04:05.978040 30839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:04:05.978451 30839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:05.982990 30839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:05.983731 30839 leader_election.cc:302] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304aa96dc0514b9dace7b3f2ef7fcf14; no voters:
I20260812 03:04:05.985436 30839 leader_election.cc:288] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:04:05.985852 30849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:05.988190 30849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 LEADER]: Becoming Leader. State: Replica: 304aa96dc0514b9dace7b3f2ef7fcf14, State: Running, Role: LEADER
I20260812 03:04:05.988956 30849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:05.989616 30839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:04:06.002508 30852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304aa96dc0514b9dace7b3f2ef7fcf14. Latest consensus state: current_term: 1 leader_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } } }
I20260812 03:04:06.003432 30852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:06.005293 30850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } } }
I20260812 03:04:06.006465 30850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:06.008672 30870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:04:06.021507 30870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:04:06.022874 30839 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 03:04:06.065459 30870 catalog_manager.cc:1347] Generated new cluster ID: 54596f212eaa4794acf0e5cf1e932783
I20260812 03:04:06.065948 30870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:04:06.108275 30870 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:04:06.111100 30870 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:04:06.132530 30870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Generated new TSK 0
I20260812 03:04:06.133917 30870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:04:06.915284 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41899
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 03:04:07 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786503847, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 03:04:07.383296 30916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:07.384052 30916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:07.384828 30916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:07.441124 30916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:07.441740 30916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:07.443480 30916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:04:07.508975 30916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.118.84:45357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:07.510772 30916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:07.512290 30916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 03:04:07.571400 30916 server_base.cc:1034] running on GCE node
Aug 12 03:04:07 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503847, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
Aug 12 03:04:08 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503847, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
W20260812 03:04:07.572597 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:08.572309 30916 thread.cc:536] Waited for 1000ms trying to join with AWS (tid 30928)
W20260812 03:04:07.572330 30935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:07.573092 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:08.995258 30916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:08.998118 30916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:08.999677 30916 hybrid_clock.cc:648] HybridClock initialized: now 1786503848999635 us; error 49 us; skew 500 ppm
I20260812 03:04:09.000556 30916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:09 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503849, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:09.006773 30916 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:04:09.012995 30916 webserver.cc:460] Webserver started at http://127.10.118.65:34639/ using document root <none> and password file <none>
I20260812 03:04:09.013942 30916 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:09.014190 30916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:09.014642 30916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:04:09 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503849, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:09.038623 30916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "4ff1570ef9c64db6acbf80536a29633e"
format_stamp: "Formatted at 2026-08-12 03:04:09 on dist-test-slave-zpvx"
server_key: "4efa5736afb557270439926f4ea28033"
server_key_iv: "f6b0526d1cf38b4c371670ef023de85c"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:09.040405 30916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "4ff1570ef9c64db6acbf80536a29633e"
format_stamp: "Formatted at 2026-08-12 03:04:09 on dist-test-slave-zpvx"
server_key: "4efa5736afb557270439926f4ea28033"
server_key_iv: "f6b0526d1cf38b4c371670ef023de85c"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:09.050967 30916 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260812 03:04:09.076586 30968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:09.078197 30916 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 03:04:09.078635 30916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "4ff1570ef9c64db6acbf80536a29633e"
format_stamp: "Formatted at 2026-08-12 03:04:09 on dist-test-slave-zpvx"
server_key: "4efa5736afb557270439926f4ea28033"
server_key_iv: "f6b0526d1cf38b4c371670ef023de85c"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:09.079144 30916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:09.145047 30916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:09.150652 30916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:09.151185 30916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:09.154460 30916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:09.158736 30916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:09.158962 30916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:09.159222 30916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:09.159404 30916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:09.301496 30916 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:41623
I20260812 03:04:09.301604 31179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:41623 every 8 connection(s)
I20260812 03:04:09.304154 30916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:04:09.312192 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 30916
I20260812 03:04:09.312815 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:04:09.314477 10713 external_mini_cluster.cc:1428] Setting key 64d07d1c859f7d0d2e13b8456488aa19
Aug 12 03:04:09 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503849, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
I20260812 03:04:09.345314 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:09.309016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:60791 (local address 127.10.118.126:41899)
0812 03:04:09.310726 (+ 1710us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:09.310750 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:09.311712 (+ 962us) server_negotiation.cc:407] Connection header received
0812 03:04:09.318494 (+ 6782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:09.318521 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:09.318780 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:09.318929 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:09.322027 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:09.323120 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:09.326516 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:09.327478 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:09.327781 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:09.334127 (+ 6346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:09.334156 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:09.334175 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:09.334270 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:09.338042 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:09.339634 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:09.339654 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:09.339667 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:09.339860 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:09.340671 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:09.340690 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:09.340704 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:09.341207 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:09.341458 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:09.342509 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0812 03:04:09.342980 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1116,"thread_start_us":738,"threads_started":1}
I20260812 03:04:09.346086 31187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:09.308745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41899 (local address 127.10.118.65:60791)
0812 03:04:09.311405 (+ 2660us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:09.311503 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:09.318152 (+ 6649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:09.319260 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:09.319297 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:09.320351 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:09.321684 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:09.321756 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:09.323420 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:09.323459 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:09.326229 (+ 2770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:09.326292 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:09.327837 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:09.327879 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:09.328492 (+ 613us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:09.329588 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:09.329691 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:09.333759 (+ 4068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:09.338261 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:09.338310 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:09.338353 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:09.339417 (+ 1064us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:09.340070 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:09.340089 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:09.340114 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:09.340459 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:09.341418 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:09.341463 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:09.341912 (+ 449us) client_negotiation.cc:769] Sending connection context
0812 03:04:09.342268 (+ 356us) client_negotiation.cc:240] Negotiation successful
0812 03:04:09.342945 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2132,"thread_start_us":1378,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:04:09.348546 31184 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41899
I20260812 03:04:09.348946 31184 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:09.350026 31184 heartbeater.cc:511] Master 127.10.118.126:41899 requested a full tablet report, sending...
I20260812 03:04:09.353117 30773 ts_manager.cc:194] Registered new tserver with Master: 4ff1570ef9c64db6acbf80536a29633e (127.10.118.65:41623)
I20260812 03:04:09.355837 30773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:60791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:04:09.376894 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41899
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:09.705940 31192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:09.706486 31192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:09.707199 31192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:09.738183 31192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:09.738574 31192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:09.739514 31192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:04:09.774927 31192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.118.84:45357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:09.776163 31192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:09.777372 31192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:09.796341 31200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:10.359597 31184 heartbeater.cc:503] Master 127.10.118.126:41899 was elected leader, sending a full tablet report...
W20260812 03:04:09.796416 31199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:09.796456 31205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:09.798075 31192 server_base.cc:1034] running on GCE node
I20260812 03:04:10.978703 31192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:10.981695 31192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:10.983130 31192 hybrid_clock.cc:648] HybridClock initialized: now 1786503850983103 us; error 50 us; skew 500 ppm
I20260812 03:04:10.983932 31192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:10 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503850, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:10.995370 31192 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:04:11.000862 31192 webserver.cc:460] Webserver started at http://127.10.118.66:37775/ using document root <none> and password file <none>
I20260812 03:04:11.001940 31192 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:11.002177 31192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:11.002663 31192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:04:11 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503850, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:11.025064 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "e95d6fcfcc1846ee9fe85f403203a59e"
format_stamp: "Formatted at 2026-08-12 03:04:11 on dist-test-slave-zpvx"
server_key: "fea605ecafb2dc26824197187a0b84a6"
server_key_iv: "51cebae485d13ce675ccb2ce0a463190"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:11.026304 31192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "e95d6fcfcc1846ee9fe85f403203a59e"
format_stamp: "Formatted at 2026-08-12 03:04:11 on dist-test-slave-zpvx"
server_key: "fea605ecafb2dc26824197187a0b84a6"
server_key_iv: "51cebae485d13ce675ccb2ce0a463190"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:11.034353 31192 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20260812 03:04:11.052194 31218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:11.053489 31192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 03:04:11.053838 31192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "e95d6fcfcc1846ee9fe85f403203a59e"
format_stamp: "Formatted at 2026-08-12 03:04:11 on dist-test-slave-zpvx"
server_key: "fea605ecafb2dc26824197187a0b84a6"
server_key_iv: "51cebae485d13ce675ccb2ce0a463190"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:11.054180 31192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:11.098296 31192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:11.104236 31192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:11.104797 31192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:11.107546 31192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:11.112483 31192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:11.112715 31192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:11.112982 31192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:11.113157 31192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:11.293500 31192 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:36541
I20260812 03:04:11.293623 31411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:36541 every 8 connection(s)
I20260812 03:04:11.296149 31192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:04:11.305789 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 31192
I20260812 03:04:11.306340 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:04:11.307653 10713 external_mini_cluster.cc:1428] Setting key d48c2fc68598f60ca86bbd325021ae8c
Aug 12 03:04:11 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503850, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:04:11.346843 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:11.304720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:39911 (local address 127.10.118.126:41899)
0812 03:04:11.321946 (+ 17226us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:11.321975 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:11.322034 (+ 59us) server_negotiation.cc:407] Connection header received
0812 03:04:11.322208 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:11.322242 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:11.322508 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:11.322730 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:11.325878 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:11.327129 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:11.329439 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:11.330076 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:11.330372 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:11.335437 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:11.335461 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:11.335477 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:11.335575 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:11.339474 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:11.340714 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:11.340735 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:11.340751 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:11.340966 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:11.341731 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:11.341750 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:11.341794 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:11.342382 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:11.342638 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:11.343967 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0812 03:04:11.344505 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16683,"thread_start_us":16379,"threads_started":1}
I20260812 03:04:11.347545 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:11.302892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41899 (local address 127.10.118.66:39911)
0812 03:04:11.305933 (+ 3041us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:11.306029 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:11.312746 (+ 6717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:11.323108 (+ 10362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:11.323149 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:11.324230 (+ 1081us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:11.325502 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:11.325566 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:11.327398 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:11.327430 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:11.329173 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:11.329212 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:11.330414 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:11.330451 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:11.330924 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:11.331871 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:11.331935 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:11.335082 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:11.339705 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:11.339741 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:11.339765 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:11.340504 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:11.341189 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:11.341215 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:11.341241 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:11.341516 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:11.342594 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:11.342640 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:11.343226 (+ 586us) client_negotiation.cc:769] Sending connection context
0812 03:04:11.343679 (+ 453us) client_negotiation.cc:240] Negotiation successful
0812 03:04:11.344448 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2438,"thread_start_us":4972,"threads_started":1}
I20260812 03:04:11.349962 31413 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41899
I20260812 03:04:11.350379 31413 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:11.351332 31413 heartbeater.cc:511] Master 127.10.118.126:41899 requested a full tablet report, sending...
I20260812 03:04:11.353629 30773 ts_manager.cc:194] Registered new tserver with Master: e95d6fcfcc1846ee9fe85f403203a59e (127.10.118.66:36541)
I20260812 03:04:11.355799 30773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:39911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:04:11.377643 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41899
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:11.698679 31422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:11.699216 31422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:11.699888 31422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:11.730894 31422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:11.731282 31422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:11.732260 31422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:04:11.768033 31422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.118.84:45357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:11.769276 31422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:11.770320 31422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 03:04:12.360219 31413 heartbeater.cc:503] Master 127.10.118.126:41899 was elected leader, sending a full tablet report...
I20260812 03:04:11.791332 31422 server_base.cc:1034] running on GCE node
W20260812 03:04:11.790135 31434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:11.789724 31438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:11.789764 31435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:12.990608 31422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:12.992872 31422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:12.994222 31422 hybrid_clock.cc:648] HybridClock initialized: now 1786503852994201 us; error 49 us; skew 500 ppm
I20260812 03:04:12.995141 31422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:12 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503852, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:13.001231 31422 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:04:13.006443 31422 webserver.cc:460] Webserver started at http://127.10.118.67:33221/ using document root <none> and password file <none>
I20260812 03:04:13.007342 31422 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:13.007525 31422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:13.007997 31422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 03:04:13 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503852, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:13.031466 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "a3a8b256cac94bf2a133eddfc743db70"
format_stamp: "Formatted at 2026-08-12 03:04:13 on dist-test-slave-zpvx"
server_key: "f09bf9ea3e30ef3d0ff020f439216bca"
server_key_iv: "84cf716192b3b64a876bfcf777a093b5"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:13.033228 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "a3a8b256cac94bf2a133eddfc743db70"
format_stamp: "Formatted at 2026-08-12 03:04:13 on dist-test-slave-zpvx"
server_key: "f09bf9ea3e30ef3d0ff020f439216bca"
server_key_iv: "84cf716192b3b64a876bfcf777a093b5"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:13.043071 31422 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260812 03:04:13.064574 31451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:13.065800 31422 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 03:04:13.066159 31422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "a3a8b256cac94bf2a133eddfc743db70"
format_stamp: "Formatted at 2026-08-12 03:04:13 on dist-test-slave-zpvx"
server_key: "f09bf9ea3e30ef3d0ff020f439216bca"
server_key_iv: "84cf716192b3b64a876bfcf777a093b5"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:13.066589 31422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:13.103819 31422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:13.109205 31422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:13.109678 31422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:13.112150 31422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:13.116096 31422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:13.116336 31422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:13.116653 31422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:13.116829 31422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:13.256460 31422 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:36691
I20260812 03:04:13.256577 31639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:36691 every 8 connection(s)
I20260812 03:04:13.259029 31422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:04:13.263363 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 31422
I20260812 03:04:13.263940 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:04:13.265622 10713 external_mini_cluster.cc:1428] Setting key dab1d3c0141ac51725da0ade130b41e0
Aug 12 03:04:13 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503852, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:13.296257 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:13.264687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:55517 (local address 127.10.118.126:41899)
0812 03:04:13.265902 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:13.265927 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:13.267541 (+ 1614us) server_negotiation.cc:407] Connection header received
0812 03:04:13.272816 (+ 5275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:13.272836 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:13.273005 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:13.273146 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:13.275471 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:13.276458 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:13.278529 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:13.279120 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:13.279328 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:13.284547 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:13.284585 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:13.284610 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:13.284714 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:13.288982 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:13.290974 (+ 1992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:13.291009 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:13.291024 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:13.291228 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:13.292072 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:13.292093 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:13.292107 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:13.292696 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:13.292883 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:13.294223 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0812 03:04:13.294567 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":630,"threads_started":1}
I20260812 03:04:13.297873 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:13.264075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41899 (local address 127.10.118.67:55517)
0812 03:04:13.267273 (+ 3198us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:13.267369 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:13.272500 (+ 5131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:13.273448 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:13.273473 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:13.274249 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:13.275222 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:13.275264 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:13.276679 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:13.276700 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:13.278315 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:13.278354 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:13.279401 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:13.279423 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:13.279774 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:13.280708 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:13.280780 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:13.284141 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:13.289278 (+ 5137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:13.289316 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:13.289351 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:13.290422 (+ 1071us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:13.291417 (+ 995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:13.291451 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:13.291476 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:13.291873 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:13.292912 (+ 1039us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:13.292953 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:13.293579 (+ 626us) client_negotiation.cc:769] Sending connection context
0812 03:04:13.294033 (+ 454us) client_negotiation.cc:240] Negotiation successful
0812 03:04:13.294754 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2632,"thread_start_us":1900,"threads_started":1}
I20260812 03:04:13.300851 31642 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41899
I20260812 03:04:13.301265 31642 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:13.302554 31642 heartbeater.cc:511] Master 127.10.118.126:41899 requested a full tablet report, sending...
I20260812 03:04:13.305512 30772 ts_manager.cc:194] Registered new tserver with Master: a3a8b256cac94bf2a133eddfc743db70 (127.10.118.67:36691)
I20260812 03:04:13.307797 30772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:55517
I20260812 03:04:13.315080 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:04:13.325881 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 30916
I20260812 03:04:13.384621 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 31192
I20260812 03:04:13.416148 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 31422
I20260812 03:04:13.448472 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 30725
I20260812 03:04:13.498090 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41899
--webserver_interface=127.10.118.126
--webserver_port=40001
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:41899
--ranger_config_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:13.805797 31647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:13.806380 31647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:13.806917 31647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:13.836818 31647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:04:13.837173 31647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:13.837419 31647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:13.837611 31647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:04:13.837795 31647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:04:13.873008 31647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:41899
--ranger_config_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:41899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=40001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:13.874099 31647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:13.875053 31647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:13.892310 31660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:13.892349 31663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:13.892328 31669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:13.893949 31647 server_base.cc:1034] running on GCE node
I20260812 03:04:15.075527 31647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:15.078311 31647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:15.079722 31647 hybrid_clock.cc:648] HybridClock initialized: now 1786503855079702 us; error 61 us; skew 500 ppm
I20260812 03:04:15.080664 31647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:15 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503855, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:15.086675 31647 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:04:15.091693 31647 webserver.cc:460] Webserver started at http://127.10.118.126:40001/ using document root <none> and password file <none>
I20260812 03:04:15.092666 31647 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:15.092933 31647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 03:04:15 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503855, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:15.165449 31647 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 03:04:15.169901 31684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:15.170971 31647 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:04:15.171357 31647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "304aa96dc0514b9dace7b3f2ef7fcf14"
format_stamp: "Formatted at 2026-08-12 03:04:05 on dist-test-slave-zpvx"
server_key: "2fb8570bff42cc94341aefb8e6d594c5"
server_key_iv: "89f2618fd6b14a295d7dcc0b3d33d4e2"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:15.173816 31647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:15.227344 31647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:15.232770 31647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:15.233287 31647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:15.302762 31647 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:41899
I20260812 03:04:15.302904 31782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:41899 every 8 connection(s)
I20260812 03:04:15.305665 31647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:04:15.314038 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 31647
I20260812 03:04:15.315968 31786 sys_catalog.cc:263] Verifying existing consensus state
I20260812 03:04:15.315892 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:41623
--local_ip_for_outbound_sockets=127.10.118.65
--tserver_master_addrs=127.10.118.126:41899
--webserver_port=34639
--webserver_interface=127.10.118.65
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 03:04:15.321326 31786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Bootstrap starting.
I20260812 03:04:15.373104 31786 log.cc:824] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:15.389228 31786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 03:04:15.390030 31786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Bootstrap complete.
I20260812 03:04:15.413335 31786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:15.415532 31786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304aa96dc0514b9dace7b3f2ef7fcf14, State: Initialized, Role: FOLLOWER
I20260812 03:04:15.416467 31786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:15.416976 31786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:04:15.417359 31786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:04:15.417687 31786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 1 FOLLOWER]: Advancing to term 2
I20260812 03:04:15.423705 31786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:15.424474 31786 leader_election.cc:302] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304aa96dc0514b9dace7b3f2ef7fcf14; no voters:
I20260812 03:04:15.426297 31786 leader_election.cc:288] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 03:04:15.426862 31790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 03:04:15.430083 31790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [term 2 LEADER]: Becoming Leader. State: Replica: 304aa96dc0514b9dace7b3f2ef7fcf14, State: Running, Role: LEADER
I20260812 03:04:15.430995 31790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } }
I20260812 03:04:15.431409 31786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:04:15.443063 31797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304aa96dc0514b9dace7b3f2ef7fcf14. Latest consensus state: current_term: 2 leader_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } } }
I20260812 03:04:15.443864 31797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:15.443053 31793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304aa96dc0514b9dace7b3f2ef7fcf14" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 41899 } } }
I20260812 03:04:15.445618 31793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14 [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:15.450462 31806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:04:15.463094 31806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:04:15.467741 31806 catalog_manager.cc:1259] Loaded cluster ID: 54596f212eaa4794acf0e5cf1e932783
I20260812 03:04:15.468075 31806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:04:15.476104 31806 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:04:15.480723 31806 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Loaded TSK: 0
I20260812 03:04:15.482563 31806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:04:15.493708 31786 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260812 03:04:15.712754 31788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:15.713526 31788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:15.714583 31788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:15.768245 31788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:15.768841 31788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:15.770511 31788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:04:15.831614 31788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:41623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=34639
--tserver_master_addrs=127.10.118.126:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:15.833259 31788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:15.834743 31788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:15.869524 31845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:15.880971 31848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:17.258437 31844 debug-util.cc:398] Leaking SignalData structure 0x7b0800014ca0 after lost signal to thread 31788
W20260812 03:04:17.375649 31849 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260812 03:04:17.376331 31788 thread.cc:636] OpenStack (cloud detector) Time spent creating pthread: real 1.506s user 0.475s sys 0.919s
W20260812 03:04:17.379297 31788 thread.cc:602] OpenStack (cloud detector) Time spent starting thread: real 1.509s user 0.476s sys 0.920s
W20260812 03:04:17.380034 31850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:17.380736 31788 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:04:17.382445 31788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:17.391211 31788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:17.392778 31788 hybrid_clock.cc:648] HybridClock initialized: now 1786503857392695 us; error 91 us; skew 500 ppm
I20260812 03:04:17.394356 31788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:17 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503857, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:17.403403 31788 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:04:17.449394 31788 webserver.cc:460] Webserver started at http://127.10.118.65:34639/ using document root <none> and password file <none>
I20260812 03:04:17.450912 31788 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:17.451308 31788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 03:04:17 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503857, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:17.495278 31788 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.005s sys 0.001s
I20260812 03:04:17.501380 31887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:17.502704 31788 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 03:04:17.503048 31788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "4ff1570ef9c64db6acbf80536a29633e"
format_stamp: "Formatted at 2026-08-12 03:04:09 on dist-test-slave-zpvx"
server_key: "4efa5736afb557270439926f4ea28033"
server_key_iv: "f6b0526d1cf38b4c371670ef023de85c"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:17.505455 31788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:17.552597 31788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:17.559882 31788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:17.560552 31788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:17.565037 31788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:17.571038 31788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:17.571382 31788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 03:04:17.571810 31788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:17.572108 31788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 03:04:17 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786503857, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:18.031533 31788 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:41623
I20260812 03:04:18.031893 32077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:41623 every 8 connection(s)
I20260812 03:04:18.037261 31788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:04:18.048192 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 31788
I20260812 03:04:18.050711 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:36541
--local_ip_for_outbound_sockets=127.10.118.66
--tserver_master_addrs=127.10.118.126:41899
--webserver_port=37775
--webserver_interface=127.10.118.66
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 03:04:18 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503857, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:18.132838 32087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:18.052211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:41911 (local address 127.10.118.126:41899)
0812 03:04:18.073073 (+ 20862us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:18.073125 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:18.076704 (+ 3579us) server_negotiation.cc:407] Connection header received
0812 03:04:18.093804 (+ 17100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:18.093851 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:18.094334 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:18.094776 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:18.098743 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:18.101403 (+ 2660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:18.104718 (+ 3315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:18.106100 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:18.106416 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:18.115467 (+ 9051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:18.115511 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:18.115553 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:18.115718 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:18.122056 (+ 6338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:18.123727 (+ 1671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:18.123759 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:18.123792 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:18.124169 (+ 377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:18.125147 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:18.125175 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:18.125196 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:18.126197 (+ 1001us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:18.126494 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:18.128137 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0812 03:04:18.128780 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20169,"thread_start_us":20662,"threads_started":1}
I20260812 03:04:18.136641 32086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:18.049939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41899 (local address 127.10.118.65:41911)
0812 03:04:18.074667 (+ 24728us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:18.074773 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:18.092881 (+ 18108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:18.095957 (+ 3076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:18.095996 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:18.097066 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:18.098331 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:18.098393 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:18.101803 (+ 3410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:18.101839 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:18.104401 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:18.104459 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:18.107135 (+ 2676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:18.107169 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:18.107668 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:18.108801 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:18.108894 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:18.114957 (+ 6063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:18.122355 (+ 7398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:18.122389 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:18.122427 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:18.123442 (+ 1015us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:18.124464 (+ 1022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:18.124492 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:18.124521 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:18.124896 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:18.126678 (+ 1782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:18.126733 (+ 55us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:18.127318 (+ 585us) client_negotiation.cc:769] Sending connection context
0812 03:04:18.131419 (+ 4101us) client_negotiation.cc:240] Negotiation successful
0812 03:04:18.132126 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24095,"thread_start_us":26633,"threads_started":1}
I20260812 03:04:18.140586 32078 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41899
I20260812 03:04:18.141067 32078 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:18.142247 32078 heartbeater.cc:511] Master 127.10.118.126:41899 requested a full tablet report, sending...
I20260812 03:04:18.145833 31718 ts_manager.cc:194] Registered new tserver with Master: 4ff1570ef9c64db6acbf80536a29633e (127.10.118.65:41623)
I20260812 03:04:18.148963 31718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:41911
Aug 12 03:04:18 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503857, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
W20260812 03:04:18.587725 32096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:18.588512 32096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:18.589733 32096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:18.646688 32096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:18.647286 32096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:18.648975 32096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:04:18.711284 32096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:36541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=37775
--tserver_master_addrs=127.10.118.126:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:18.712886 32096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:18.714316 32096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:18.745272 32107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 12 03:04:18 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503857, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:19.157079 32078 heartbeater.cc:503] Master 127.10.118.126:41899 was elected leader, sending a full tablet report...
W20260812 03:04:20.136446 32106 debug-util.cc:398] Leaking SignalData structure 0x7b0800014ca0 after lost signal to thread 32096
W20260812 03:04:20.186794 32096 thread.cc:636] GCE (cloud detector) Time spent creating pthread: real 1.444s user 0.426s sys 0.823s
W20260812 03:04:18.745280 32108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:20.187232 32096 thread.cc:602] GCE (cloud detector) Time spent starting thread: real 1.445s user 0.426s sys 0.823s
W20260812 03:04:20.197935 32128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:20.198103 32096 server_base.cc:1034] running on GCE node
I20260812 03:04:20.199358 32096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:20.201582 32096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:20.202958 32096 hybrid_clock.cc:648] HybridClock initialized: now 1786503860202928 us; error 32 us; skew 500 ppm
I20260812 03:04:20.203750 32096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:20 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503860, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:20.209950 32096 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:04:20.215214 32096 webserver.cc:460] Webserver started at http://127.10.118.66:37775/ using document root <none> and password file <none>
I20260812 03:04:20.216306 32096 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:20.216627 32096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 03:04:20 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503860, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:20.262737 32096 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.003s
I20260812 03:04:20.267455 32137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:20.268786 32096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 03:04:20.269150 32096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "e95d6fcfcc1846ee9fe85f403203a59e"
format_stamp: "Formatted at 2026-08-12 03:04:11 on dist-test-slave-zpvx"
server_key: "fea605ecafb2dc26824197187a0b84a6"
server_key_iv: "51cebae485d13ce675ccb2ce0a463190"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:20.271556 32096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:20.313331 32096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:20.320410 32096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:20.320955 32096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:20.324281 32096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:20.329618 32096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:20.329905 32096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:20.330204 32096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:20.330430 32096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:20.468500 32096 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:36541
I20260812 03:04:20.468797 32325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:36541 every 8 connection(s)
I20260812 03:04:20.471519 32096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:04:20.481221 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 32096
I20260812 03:04:20.483646 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.118.84:45357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:36691
--local_ip_for_outbound_sockets=127.10.118.67
--tserver_master_addrs=127.10.118.126:41899
--webserver_port=33221
--webserver_interface=127.10.118.67
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 03:04:20 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503860, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:20.510538 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:20.475929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:44647 (local address 127.10.118.126:41899)
0812 03:04:20.477736 (+ 1807us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:20.477765 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:20.479012 (+ 1247us) server_negotiation.cc:407] Connection header received
0812 03:04:20.485474 (+ 6462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:20.485495 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:20.485673 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:20.485816 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:20.488991 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:20.489931 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:20.493335 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:20.494467 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:20.494783 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:20.499714 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:20.499738 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:20.499754 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:20.499850 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:20.503661 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:20.504913 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:20.504936 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:20.504949 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:20.505139 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:20.505907 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:20.505926 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:20.505939 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:20.506426 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:20.506684 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:20.507736 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0812 03:04:20.508195 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":891,"threads_started":1}
I20260812 03:04:20.511315 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:20.476211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41899 (local address 127.10.118.66:44647)
0812 03:04:20.478751 (+ 2540us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:20.478821 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:20.485082 (+ 6261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:20.486177 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:20.486221 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:20.487343 (+ 1122us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:20.488671 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:20.488748 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:20.490194 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:20.490232 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:20.493045 (+ 2813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:20.493109 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:20.494869 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:20.494891 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:20.495299 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:20.496322 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:20.496448 (+ 126us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:20.499359 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:20.503862 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:20.503907 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:20.503932 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:20.504703 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:20.505344 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:20.505380 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:20.505400 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:20.505701 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:20.506638 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:20.506680 (+ 42us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:20.507137 (+ 457us) client_negotiation.cc:769] Sending connection context
0812 03:04:20.507491 (+ 354us) client_negotiation.cc:240] Negotiation successful
0812 03:04:20.508188 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2087,"thread_start_us":1374,"threads_started":1}
I20260812 03:04:20.513396 32328 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41899
I20260812 03:04:20.513849 32328 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:20.514806 32328 heartbeater.cc:511] Master 127.10.118.126:41899 requested a full tablet report, sending...
I20260812 03:04:20.517292 31718 ts_manager.cc:194] Registered new tserver with Master: e95d6fcfcc1846ee9fe85f403203a59e (127.10.118.66:36541)
I20260812 03:04:20.519299 31718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:44647
W20260812 03:04:20.807142 32332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:20.807632 32332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:20.808223 32332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:20.839316 32332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:20.839706 32332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:20.840700 32332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:04:20.876019 32332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.118.84:45357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:36691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=33221
--tserver_master_addrs=127.10.118.126:41899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:20.877270 32332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:20.878352 32332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:20.897295 32343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:21.522830 32328 heartbeater.cc:503] Master 127.10.118.126:41899 was elected leader, sending a full tablet report...
W20260812 03:04:20.897382 32346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:20.898424 32341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:22.085366 32345 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:04:22.085527 32332 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:04:22.089135 32332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:22.092154 32332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:22.093685 32332 hybrid_clock.cc:648] HybridClock initialized: now 1786503862093623 us; error 87 us; skew 500 ppm
I20260812 03:04:22.094578 32332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:22 dist-test-slave-zpvx krb5kdc[29952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503862, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:22.104336 32332 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:04:22.110291 32332 webserver.cc:460] Webserver started at http://127.10.118.67:33221/ using document root <none> and password file <none>
I20260812 03:04:22.111243 32332 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:22.111505 32332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 03:04:22 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503862, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for HTTP/127.10.118.84@KRBTEST.COM
I20260812 03:04:22.139511 32332 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 03:04:22.144351 32362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:22.145591 32332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:04:22.145928 32332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "a3a8b256cac94bf2a133eddfc743db70"
format_stamp: "Formatted at 2026-08-12 03:04:13 on dist-test-slave-zpvx"
server_key: "f09bf9ea3e30ef3d0ff020f439216bca"
server_key_iv: "84cf716192b3b64a876bfcf777a093b5"
server_key_version: "kuduclusterkey@0"
I20260812 03:04:22.148289 32332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:22.197830 32332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:22.203374 32332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:22.203848 32332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:22.206691 32332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:22.211807 32332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:22.212044 32332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:22.212299 32332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:22.212530 32332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 03:04:22.358359 32332 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:36691
I20260812 03:04:22.358544 32565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:36691 every 8 connection(s)
I20260812 03:04:22.361413 32332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:04:22.367070 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 32332
Aug 12 03:04:22 dist-test-slave-zpvx krb5kdc[29952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503862, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:22.394033 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.368982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.10.118.126:41899)
0812 03:04:22.370626 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.370647 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.371597 (+ 950us) server_negotiation.cc:407] Connection header received
0812 03:04:22.372144 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.372170 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.372344 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.372538 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:22.376045 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.377031 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.379095 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.379859 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.380157 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.383685 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:22.383706 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:22.383728 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:22.383831 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:22.387750 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:22.388900 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:22.388921 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:22.388935 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:22.389159 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:22.389817 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:22.389837 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:22.389850 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:22.390502 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:22.390931 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.391655 (+ 724us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.392307 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":1156,"thread_start_us":847,"threads_started":1}
I20260812 03:04:22.399658 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.365591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:55643 (local address 127.10.118.126:41899)
0812 03:04:22.368959 (+ 3368us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.368982 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.369452 (+ 470us) server_negotiation.cc:407] Connection header received
0812 03:04:22.376486 (+ 7034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.376521 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.376691 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.376870 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:22.379012 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.380162 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.382417 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.383015 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.383292 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.388356 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:22.388431 (+ 75us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:22.388459 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:22.388559 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:22.393167 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:22.394567 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:22.394599 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:22.394629 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:22.394839 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:22.395685 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:22.395710 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:22.395724 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:22.396329 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:22.396566 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.397705 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.398215 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":2884,"thread_start_us":585,"threads_started":1}
I20260812 03:04:22.400528 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.365791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:41899 (local address 127.10.118.67:55643)
0812 03:04:22.369124 (+ 3333us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.369240 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:22.376143 (+ 6903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:22.377140 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:22.377173 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:22.377907 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:22.378738 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.378778 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.380432 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.380452 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.382159 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.382207 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.383355 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.383388 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.383840 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.384715 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:22.384832 (+ 117us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:22.388016 (+ 3184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:22.393380 (+ 5364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:22.393416 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:22.393450 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:22.394301 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:22.395060 (+ 759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:22.395082 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:22.395099 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:22.395453 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:22.396576 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:22.396617 (+ 41us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:22.397103 (+ 486us) client_negotiation.cc:769] Sending connection context
0812 03:04:22.397410 (+ 307us) client_negotiation.cc:240] Negotiation successful
0812 03:04:22.397963 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2493,"thread_start_us":1746,"threads_started":1}
I20260812 03:04:22.402248 32566 heartbeater.cc:347] Connected to a master server at 127.10.118.126:41899
I20260812 03:04:22.402647 32566 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:22.403546 32566 heartbeater.cc:511] Master 127.10.118.126:41899 requested a full tablet report, sending...
I20260812 03:04:22.405637 31718 ts_manager.cc:194] Registered new tserver with Master: a3a8b256cac94bf2a133eddfc743db70 (127.10.118.67:36691)
I20260812 03:04:22.407027 31718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:55643
I20260812 03:04:22.412216 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:04:22.438501 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.418480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.10.118.126:41899)
0812 03:04:22.419056 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.419100 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.419945 (+ 845us) server_negotiation.cc:407] Connection header received
0812 03:04:22.420254 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.420273 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.420494 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.420624 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:22.422798 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.423649 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.425479 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.426038 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.426248 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.429852 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:22.429871 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:22.429887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:22.429983 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:22.433602 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:22.434560 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:22.434580 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:22.434595 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:22.434780 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:22.435353 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:22.435374 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:22.435387 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:22.435876 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:22.436162 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.436756 (+ 594us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.437056 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 03:04:22.445240 31718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:04:22.465667 31718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:04:22.524294 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.500473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:41623 (local address 127.0.0.1:42418)
0812 03:04:22.503443 (+ 2970us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.503527 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:22.503853 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:22.506803 (+ 2950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:22.506841 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:22.506948 (+ 107us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:22.507988 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.508039 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.512016 (+ 3977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.512066 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.515737 (+ 3671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.515814 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.519187 (+ 3373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.519238 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.519895 (+ 657us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.520143 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 03:04:22.520424 (+ 281us) client_negotiation.cc:240] Negotiation successful
0812 03:04:22.522208 (+ 1784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2088,"mutex_wait_us":1197,"thread_start_us":2205,"threads_started":1}
I20260812 03:04:22.524932 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.500671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.10.118.65:41623)
0812 03:04:22.505535 (+ 4864us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.505578 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.505651 (+ 73us) server_negotiation.cc:407] Connection header received
0812 03:04:22.505884 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.505928 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.506394 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.507782 (+ 1388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:22.508971 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.511650 (+ 2679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.516331 (+ 4681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.518664 (+ 2333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.521093 (+ 2429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.521744 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.521930 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.523317 (+ 1387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4284,"thread_start_us":4454,"threads_started":1}
I20260812 03:04:22.530154 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.503824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:36541 (local address 127.0.0.1:54428)
0812 03:04:22.507633 (+ 3809us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.508336 (+ 703us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:22.508669 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:22.510353 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:22.510388 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:22.510507 (+ 119us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:22.512791 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.512862 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.517261 (+ 4399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.517313 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.522277 (+ 4964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.522340 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.525848 (+ 3508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.525883 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.526383 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.526585 (+ 202us) client_negotiation.cc:769] Sending connection context
0812 03:04:22.526747 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 03:04:22.527054 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3438,"mutex_wait_us":151,"thread_start_us":3723,"threads_started":1}
I20260812 03:04:22.529739 31973 tablet_service.cc:1467] Processing CreateTablet for tablet fc4eb0eab19c4a9582663013573693b2 (DEFAULT_TABLE table=test-table [id=a93fc3a1a7984fb597dca6f1e422693f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:22.531255 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.504269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.10.118.66:36541)
0812 03:04:22.506515 (+ 2246us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.506547 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.508513 (+ 1966us) server_negotiation.cc:407] Connection header received
0812 03:04:22.509024 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.509071 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.509434 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.509631 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:22.513364 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.515982 (+ 2618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.522623 (+ 6641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.524730 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.525021 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.525482 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.526873 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.527166 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1767,"thread_start_us":2107,"threads_started":1}
I20260812 03:04:22.532259 31973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4eb0eab19c4a9582663013573693b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:22.536350 32215 tablet_service.cc:1467] Processing CreateTablet for tablet fc4eb0eab19c4a9582663013573693b2 (DEFAULT_TABLE table=test-table [id=a93fc3a1a7984fb597dca6f1e422693f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:22.538359 32215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4eb0eab19c4a9582663013573693b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:22.538797 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.509009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.10.118.67:36691)
0812 03:04:22.521918 (+ 12909us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.521954 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.523795 (+ 1841us) server_negotiation.cc:407] Connection header received
0812 03:04:22.523997 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.524027 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.524413 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.524623 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:22.526985 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.529405 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.534241 (+ 4836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.535476 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.535812 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.536292 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.536720 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.536994 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12402,"thread_start_us":12772,"threads_started":1}
I20260812 03:04:22.538913 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.508672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:36691 (local address 127.0.0.1:37900)
0812 03:04:22.523266 (+ 14594us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.523335 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:22.523577 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:22.525616 (+ 2039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:22.525646 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:22.525757 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:22.526592 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.526637 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.529742 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.529773 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.533908 (+ 4135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.533983 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.535849 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.535880 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.536296 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.536469 (+ 173us) client_negotiation.cc:769] Sending connection context
0812 03:04:22.536625 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 03:04:22.536927 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14152,"mutex_wait_us":35,"thread_start_us":13972,"threads_started":1}
I20260812 03:04:22.543079 32448 tablet_service.cc:1467] Processing CreateTablet for tablet fc4eb0eab19c4a9582663013573693b2 (DEFAULT_TABLE table=test-table [id=a93fc3a1a7984fb597dca6f1e422693f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:22.544582 32448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4eb0eab19c4a9582663013573693b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:22.566469 32600 tablet_bootstrap.cc:492] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Bootstrap starting.
I20260812 03:04:22.567782 32602 tablet_bootstrap.cc:492] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Bootstrap starting.
I20260812 03:04:22.570202 32601 tablet_bootstrap.cc:492] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e: Bootstrap starting.
I20260812 03:04:22.574391 32600 tablet_bootstrap.cc:654] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:22.575306 32601 tablet_bootstrap.cc:654] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:22.575642 32602 tablet_bootstrap.cc:654] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:22.577378 32600 log.cc:824] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:22.577468 32601 log.cc:824] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:22.577983 32602 log.cc:824] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:22.584299 32600 tablet_bootstrap.cc:492] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: No bootstrap required, opened a new log
I20260812 03:04:22.584661 32601 tablet_bootstrap.cc:492] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e: No bootstrap required, opened a new log
I20260812 03:04:22.584789 32600 ts_tablet_manager.cc:1397] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260812 03:04:22.585150 32601 ts_tablet_manager.cc:1397] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260812 03:04:22.586083 32602 tablet_bootstrap.cc:492] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: No bootstrap required, opened a new log
I20260812 03:04:22.586549 32602 ts_tablet_manager.cc:1397] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.010s
I20260812 03:04:22.607165 32600 raft_consensus.cc:348] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.607874 32600 raft_consensus.cc:374] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:22.608168 32600 raft_consensus.cc:729] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a8b256cac94bf2a133eddfc743db70, State: Initialized, Role: FOLLOWER
I20260812 03:04:22.609076 32600 consensus_queue.cc:260] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.612797 32566 heartbeater.cc:503] Master 127.10.118.126:41899 was elected leader, sending a full tablet report...
I20260812 03:04:22.613368 32600 ts_tablet_manager.cc:1428] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Time spent starting tablet: real 0.028s user 0.024s sys 0.004s
W20260812 03:04:22.616089 32567 tablet.cc:2369] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:22.617282 32602 raft_consensus.cc:348] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.618239 32602 raft_consensus.cc:374] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:22.618558 32602 raft_consensus.cc:729] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95d6fcfcc1846ee9fe85f403203a59e, State: Initialized, Role: FOLLOWER
I20260812 03:04:22.619469 32602 consensus_queue.cc:260] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.620251 32601 raft_consensus.cc:348] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.621316 32601 raft_consensus.cc:374] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:22.621685 32601 raft_consensus.cc:729] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ff1570ef9c64db6acbf80536a29633e, State: Initialized, Role: FOLLOWER
I20260812 03:04:22.622844 32601 consensus_queue.cc:260] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.623711 32602 ts_tablet_manager.cc:1428] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Time spent starting tablet: real 0.037s user 0.023s sys 0.014s
I20260812 03:04:22.625425 32606 raft_consensus.cc:482] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:04:22.626003 32606 raft_consensus.cc:504] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.631652 32606 leader_election.cc:288] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ff1570ef9c64db6acbf80536a29633e (127.10.118.65:41623), e95d6fcfcc1846ee9fe85f403203a59e (127.10.118.66:36541)
I20260812 03:04:22.638372 32601 ts_tablet_manager.cc:1428] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e: Time spent starting tablet: real 0.053s user 0.027s sys 0.010s
I20260812 03:04:22.644361 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.632096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:33637 (local address 127.10.118.65:41623)
0812 03:04:22.633656 (+ 1560us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.633689 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.633743 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:04:22.633918 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.633952 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.634228 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.634408 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:22.635914 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.637243 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.640353 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.641589 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.641953 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.642220 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.642716 (+ 496us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.642981 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025}
I20260812 03:04:22.645367 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.631957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:41623 (local address 127.10.118.67:33637)
0812 03:04:22.632655 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.632701 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:22.632937 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:22.634915 (+ 1978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:22.634943 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:22.635071 (+ 128us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:22.635543 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.635593 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.637538 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.637568 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.640073 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.640122 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.641877 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.641912 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.642381 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.642488 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 03:04:22.642653 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 03:04:22.642952 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260812 03:04:22.647053 32005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4eb0eab19c4a9582663013573693b2" candidate_uuid: "a3a8b256cac94bf2a133eddfc743db70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff1570ef9c64db6acbf80536a29633e" is_pre_election: true
I20260812 03:04:22.648042 32005 raft_consensus.cc:2454] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3a8b256cac94bf2a133eddfc743db70 in term 0.
I20260812 03:04:22.649231 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.634524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:37231 (local address 127.10.118.66:36541)
0812 03:04:22.635275 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.635306 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.635425 (+ 119us) server_negotiation.cc:407] Connection header received
0812 03:04:22.635980 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.636011 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.636265 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.636422 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:22.640054 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.641146 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.644791 (+ 3645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.646395 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.646765 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.647027 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.647501 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.647785 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260812 03:04:22.649614 32613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.632840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:36541 (local address 127.10.118.67:37231)
0812 03:04:22.635257 (+ 2417us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.635318 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:22.635675 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:22.636913 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:22.636940 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:22.637040 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:22.639467 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.639518 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.641406 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.641435 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.644440 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:22.644497 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.646742 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.646776 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:22.647174 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.647268 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 03:04:22.647429 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 03:04:22.647700 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1898,"thread_start_us":1570,"threads_started":1}
I20260812 03:04:22.650269 32364 leader_election.cc:302] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ff1570ef9c64db6acbf80536a29633e, a3a8b256cac94bf2a133eddfc743db70; no voters:
I20260812 03:04:22.651034 32606 raft_consensus.cc:2781] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:04:22.651480 32606 raft_consensus.cc:482] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:04:22.651849 32606 raft_consensus.cc:3037] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:22.652297 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4eb0eab19c4a9582663013573693b2" candidate_uuid: "a3a8b256cac94bf2a133eddfc743db70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" is_pre_election: true
I20260812 03:04:22.653071 32253 raft_consensus.cc:2454] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3a8b256cac94bf2a133eddfc743db70 in term 0.
I20260812 03:04:22.657322 32606 raft_consensus.cc:504] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.658804 32606 leader_election.cc:288] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [CANDIDATE]: Term 1 election: Requested vote from peers 4ff1570ef9c64db6acbf80536a29633e (127.10.118.65:41623), e95d6fcfcc1846ee9fe85f403203a59e (127.10.118.66:36541)
I20260812 03:04:22.659701 32005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4eb0eab19c4a9582663013573693b2" candidate_uuid: "a3a8b256cac94bf2a133eddfc743db70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ff1570ef9c64db6acbf80536a29633e"
I20260812 03:04:22.659747 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4eb0eab19c4a9582663013573693b2" candidate_uuid: "a3a8b256cac94bf2a133eddfc743db70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95d6fcfcc1846ee9fe85f403203a59e"
I20260812 03:04:22.660185 32253 raft_consensus.cc:3037] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:22.660180 32005 raft_consensus.cc:3037] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:22.664837 32253 raft_consensus.cc:2454] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3a8b256cac94bf2a133eddfc743db70 in term 1.
I20260812 03:04:22.665158 32005 raft_consensus.cc:2454] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3a8b256cac94bf2a133eddfc743db70 in term 1.
I20260812 03:04:22.665657 32364 leader_election.cc:302] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3a8b256cac94bf2a133eddfc743db70, e95d6fcfcc1846ee9fe85f403203a59e; no voters:
I20260812 03:04:22.666230 32606 raft_consensus.cc:2781] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:22.667763 32606 raft_consensus.cc:686] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 1 LEADER]: Becoming Leader. State: Replica: a3a8b256cac94bf2a133eddfc743db70, State: Running, Role: LEADER
I20260812 03:04:22.668774 32606 consensus_queue.cc:237] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } }
I20260812 03:04:22.676937 31716 catalog_manager.cc:5495] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3a8b256cac94bf2a133eddfc743db70 (127.10.118.67). New cstate: current_term: 1 leader_uuid: "a3a8b256cac94bf2a133eddfc743db70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3a8b256cac94bf2a133eddfc743db70" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 36691 } health_report { overall_health: HEALTHY } } }
W20260812 03:04:22.728636 32329 tablet.cc:2369] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:22.740638 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.732183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.10.118.67:36691)
0812 03:04:22.732639 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:22.732658 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:22.732878 (+ 220us) server_negotiation.cc:407] Connection header received
0812 03:04:22.733155 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:22.733176 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:22.733480 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:22.733690 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:04:22.734380 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.735233 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.737009 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:22.737582 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:22.737779 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:22.738496 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:04:22.738967 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:04:22.739107 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:22.739327 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 03:04:22.739452 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 03:04:22.752907 32005 raft_consensus.cc:1261] T fc4eb0eab19c4a9582663013573693b2 P 4ff1570ef9c64db6acbf80536a29633e [term 1 FOLLOWER]: Refusing update from remote peer a3a8b256cac94bf2a133eddfc743db70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:04:22.753077 32253 raft_consensus.cc:1261] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 1 FOLLOWER]: Refusing update from remote peer a3a8b256cac94bf2a133eddfc743db70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:04:22.754436 32615 consensus_queue.cc:1035] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ff1570ef9c64db6acbf80536a29633e" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 41623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:04:22.755013 32606 consensus_queue.cc:1035] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e95d6fcfcc1846ee9fe85f403203a59e" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:04:22.774372 32619 mvcc.cc:205] Tried to move back new op lower bound from 7317519821819404288 to 7317519821499019264. Current Snapshot: MvccSnapshot[applied={T|T < 7317519821819404288}]
I20260812 03:04:22.779300 32622 mvcc.cc:205] Tried to move back new op lower bound from 7317519821819404288 to 7317519821499019264. Current Snapshot: MvccSnapshot[applied={T|T < 7317519821819404288}]
I20260812 03:04:22.808056 31718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:22.808598 31718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:22.817626 31718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 304aa96dc0514b9dace7b3f2ef7fcf14: Sending DeleteTablet for 3 replicas of tablet fc4eb0eab19c4a9582663013573693b2
I20260812 03:04:22.819609 31973 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4eb0eab19c4a9582663013573693b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:22 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:42418
I20260812 03:04:22.820456 32215 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4eb0eab19c4a9582663013573693b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:22 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:54428
I20260812 03:04:22.820842 32448 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4eb0eab19c4a9582663013573693b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:22 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:37900
I20260812 03:04:22.826363 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 31788
I20260812 03:04:22.844697 32635 tablet_replica.cc:330] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: stopping tablet replica
I20260812 03:04:22.845811 32635 raft_consensus.cc:2229] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:04:22.846627 32635 raft_consensus.cc:2258] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:22.848483 32636 tablet_replica.cc:330] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: stopping tablet replica
I20260812 03:04:22.849864 32636 raft_consensus.cc:2229] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:04:22.850980 32635 ts_tablet_manager.cc:1905] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:22.850979 32636 raft_consensus.cc:2258] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:22.854617 32636 ts_tablet_manager.cc:1905] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:22.866025 32635 ts_tablet_manager.cc:1918] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:22.866513 32635 log.cc:1196] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/fc4eb0eab19c4a9582663013573693b2
I20260812 03:04:22.867517 32635 ts_tablet_manager.cc:1939] T fc4eb0eab19c4a9582663013573693b2 P e95d6fcfcc1846ee9fe85f403203a59e: Deleting consensus metadata
I20260812 03:04:22.871183 31686 catalog_manager.cc:4841] TS e95d6fcfcc1846ee9fe85f403203a59e (127.10.118.66:36541): tablet fc4eb0eab19c4a9582663013573693b2 (table test-table [id=a93fc3a1a7984fb597dca6f1e422693f]) successfully deleted
I20260812 03:04:22.872103 32636 ts_tablet_manager.cc:1918] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:22.872587 32636 log.cc:1196] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/fc4eb0eab19c4a9582663013573693b2
I20260812 03:04:22.873476 32636 ts_tablet_manager.cc:1939] T fc4eb0eab19c4a9582663013573693b2 P a3a8b256cac94bf2a133eddfc743db70: Deleting consensus metadata
W20260812 03:04:22.874699 31686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:41623 (error 108)
I20260812 03:04:22.877779 31686 catalog_manager.cc:4841] TS a3a8b256cac94bf2a133eddfc743db70 (127.10.118.67:36691): tablet fc4eb0eab19c4a9582663013573693b2 (table test-table [id=a93fc3a1a7984fb597dca6f1e422693f]) successfully deleted
I20260812 03:04:22.879043 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 32096
I20260812 03:04:22.880155 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.878829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:41623 (local address 127.0.0.1:42428)
0812 03:04:22.879337 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.879594 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 03:04:22.893078 31686 catalog_manager.cc:4584] TS 4ff1570ef9c64db6acbf80536a29633e (127.10.118.65:41623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc4eb0eab19c4a9582663013573693b2: Network error: Client connection negotiation failed: client connection to 127.10.118.65:41623: connect: Connection refused (error 111)
I20260812 03:04:22.919426 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 32332
I20260812 03:04:22.948699 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.947583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:41623 (local address 127.0.0.1:42430)
0812 03:04:22.948034 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.948209 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260812 03:04:22.951093 32596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:22.950133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:41623 (local address 127.0.0.1:42442)
0812 03:04:22.950533 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:22.950707 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:41623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260812 03:04:22.955921 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 31647
I20260812 03:04:23.010800 10713 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 03:04:23.357601 10713 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 03:04:23.358151 10713 mini_ranger.cc:66] Stopping Ranger...
I20260812 03:04:23.704497 10713 mini_ranger.cc:68] Stopped Ranger
2026-08-12 03:04:23.705 UTC [30012] LOG: received smart shutdown request
2026-08-12 03:04:23.708 UTC [30012] LOG: background worker "logical replication launcher" (PID 30023) exited with exit code 1
2026-08-12 03:04:23.709 UTC [30017] LOG: shutting down
2026-08-12 03:04:23.886 UTC [30012] LOG: database system is shut down
2026-08-12T03:04:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (66709 ms)
[ RUN ] SecurityITest.IPKICACert
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1045: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.IPKICACert (13 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:04:24 dist-test-slave-zpvx krb5kdc[32655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:04:24 dist-test-slave-zpvx krb5kdc[32655](info): set up 2 sockets
Aug 12 03:04:24 dist-test-slave-zpvx krb5kdc[32655](info): commencing operation
krb5kdc: starting...
W20260812 03:04:25.676095 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.424s user 0.003s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:04:25 dist-test-slave-zpvx krb5kdc[32655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:04:25.908218 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:33425
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:39939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:33425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:26.215924 32675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:26.216615 32675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:26.217396 32675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:26.247872 32675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:04:26.248288 32675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:26.248641 32675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:26.248926 32675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:04:26.249198 32675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:04:26.284795 32675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:39939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:33425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:26.286072 32675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:26.287155 32675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:26.304569 32691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:26.304572 32686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:26.306115 32675 server_base.cc:1034] running on GCE node
W20260812 03:04:26.304839 32684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:27.485105 32675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:27.488334 32675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:27.489794 32675 hybrid_clock.cc:648] HybridClock initialized: now 1786503867489738 us; error 80 us; skew 500 ppm
I20260812 03:04:27.490586 32675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:27 dist-test-slave-zpvx krb5kdc[32655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503867, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:27.502615 32675 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:04:27.508558 32675 webserver.cc:460] Webserver started at http://127.10.118.126:43699/ using document root <none> and password file <none>
I20260812 03:04:27.509475 32675 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:27.509658 32675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:27.510078 32675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:27.515379 32675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "ac05b30e8d4841ba8d174fa493f176bd"
format_stamp: "Formatted at 2026-08-12 03:04:27 on dist-test-slave-zpvx"
server_key: "76eea2fd2039830c2787f2018bc3a98c"
server_key_iv: "c31e92e2c0836a9deb622cd49ea01a5b"
server_key_version: "encryptionkey@0"
I20260812 03:04:27.516564 32675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "ac05b30e8d4841ba8d174fa493f176bd"
format_stamp: "Formatted at 2026-08-12 03:04:27 on dist-test-slave-zpvx"
server_key: "76eea2fd2039830c2787f2018bc3a98c"
server_key_iv: "c31e92e2c0836a9deb622cd49ea01a5b"
server_key_version: "encryptionkey@0"
I20260812 03:04:27.524053 32675 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:04:27.529616 32703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:27.530658 32675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:04:27.531010 32675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "ac05b30e8d4841ba8d174fa493f176bd"
format_stamp: "Formatted at 2026-08-12 03:04:27 on dist-test-slave-zpvx"
server_key: "76eea2fd2039830c2787f2018bc3a98c"
server_key_iv: "c31e92e2c0836a9deb622cd49ea01a5b"
server_key_version: "encryptionkey@0"
I20260812 03:04:27.531392 32675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:27.562390 32675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:27.567994 32675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:27.568533 32675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:27.637053 32675 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:33425
I20260812 03:04:27.637156 319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:33425 every 8 connection(s)
I20260812 03:04:27.639747 32675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:04:27.644858 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 32675
I20260812 03:04:27.645399 320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:27.645426 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:04:27.647105 10713 external_mini_cluster.cc:1428] Setting key 5cc488d70a13a9260dadd82ba1e983a6
Aug 12 03:04:27 dist-test-slave-zpvx krb5kdc[32655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:27.669790 320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd: Bootstrap starting.
I20260812 03:04:27.675318 320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:27.677071 320 log.cc:824] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:27.681727 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:27.649715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.10.118.126:33425)
0812 03:04:27.652351 (+ 2636us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:27.652438 (+ 87us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:27.652491 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:04:27.656659 (+ 4168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:27.656696 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:27.657035 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:27.657456 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:27.660091 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:27.661835 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:27.663718 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:27.664614 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:27.664942 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:27.670756 (+ 5814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:27.670783 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:27.670811 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:27.670916 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:27.675005 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:27.676154 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:27.676178 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:27.676195 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:27.676503 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:27.677140 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:27.677169 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:27.677183 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:27.677904 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:27.678390 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:27.679111 (+ 721us) server_negotiation.cc:299] Negotiation successful
0812 03:04:27.679751 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2201,"thread_start_us":1822,"threads_started":1}
I20260812 03:04:27.682801 320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd: No bootstrap required, opened a new log
I20260812 03:04:27.686308 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:04:27.706382 320 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b30e8d4841ba8d174fa493f176bd" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33425 } }
I20260812 03:04:27.707002 320 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:27.707202 320 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac05b30e8d4841ba8d174fa493f176bd, State: Initialized, Role: FOLLOWER
I20260812 03:04:27.707795 320 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b30e8d4841ba8d174fa493f176bd" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33425 } }
I20260812 03:04:27.708279 320 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:04:27.708578 320 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:04:27.708832 320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:27.712756 320 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b30e8d4841ba8d174fa493f176bd" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33425 } }
I20260812 03:04:27.713376 320 leader_election.cc:302] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac05b30e8d4841ba8d174fa493f176bd; no voters:
I20260812 03:04:27.714967 320 leader_election.cc:288] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:04:27.715363 330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:27.717532 330 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [term 1 LEADER]: Becoming Leader. State: Replica: ac05b30e8d4841ba8d174fa493f176bd, State: Running, Role: LEADER
I20260812 03:04:27.718379 330 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b30e8d4841ba8d174fa493f176bd" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33425 } }
I20260812 03:04:27.719163 320 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:04:27.728974 334 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac05b30e8d4841ba8d174fa493f176bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b30e8d4841ba8d174fa493f176bd" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33425 } } }
I20260812 03:04:27.729079 339 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac05b30e8d4841ba8d174fa493f176bd. Latest consensus state: current_term: 1 leader_uuid: "ac05b30e8d4841ba8d174fa493f176bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac05b30e8d4841ba8d174fa493f176bd" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 33425 } } }
I20260812 03:04:27.729894 334 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:27.729980 339 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:27.734272 351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:04:27.745369 351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:04:27.761933 351 catalog_manager.cc:1347] Generated new cluster ID: e825a62377bf47c3b9407385af822c53
I20260812 03:04:27.762198 351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:04:27.801553 351 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:04:27.803661 351 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:04:27.814134 351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac05b30e8d4841ba8d174fa493f176bd: Generated new TSK 0
I20260812 03:04:27.815220 351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:04:27.865337 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 32675
2026-08-12T03:04:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (3673 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1192: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (6 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1219: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedTSK (6 ms)
[----------] 30 tests from SecurityITest (325274 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:04:27 dist-test-slave-zpvx krb5kdc[375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:04:27 dist-test-slave-zpvx krb5kdc[375](info): set up 2 sockets
Aug 12 03:04:27 dist-test-slave-zpvx krb5kdc[375](info): commencing operation
krb5kdc: starting...
W20260812 03:04:29.256748 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.323s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:04:29 dist-test-slave-zpvx krb5kdc[375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:04:29.493046 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43653
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:43653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:29.818457 399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:29.819092 399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:29.819725 399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:29.850320 399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:04:29.850688 399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:29.850998 399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:29.851269 399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:04:29.851521 399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:04:29.886834 399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:43653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:43653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:29.888046 399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:29.889137 399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:29.906450 412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:29.907529 411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:29.908281 399 server_base.cc:1034] running on GCE node
W20260812 03:04:29.906441 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:31.119226 399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:31.121937 399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:31.123294 399 hybrid_clock.cc:648] HybridClock initialized: now 1786503871123260 us; error 50 us; skew 500 ppm
I20260812 03:04:31.124132 399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:31 dist-test-slave-zpvx krb5kdc[375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503871, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:31.130519 399 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:04:31.136487 399 webserver.cc:460] Webserver started at http://127.10.118.126:37357/ using document root <none> and password file <none>
I20260812 03:04:31.137466 399 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:31.137719 399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:31.138211 399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:31.143581 399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "97b15e1740374ce2b8dff2e0ca20791e"
format_stamp: "Formatted at 2026-08-12 03:04:31 on dist-test-slave-zpvx"
server_key: "6104b7d18aac7b66eb7e79cd861c63bc"
server_key_iv: "3fcde542e2c7639b6c7be9f02d49eb7b"
server_key_version: "encryptionkey@0"
I20260812 03:04:31.144833 399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "97b15e1740374ce2b8dff2e0ca20791e"
format_stamp: "Formatted at 2026-08-12 03:04:31 on dist-test-slave-zpvx"
server_key: "6104b7d18aac7b66eb7e79cd861c63bc"
server_key_iv: "3fcde542e2c7639b6c7be9f02d49eb7b"
server_key_version: "encryptionkey@0"
I20260812 03:04:31.152272 399 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:04:31.157992 426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:31.159083 399 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:04:31.159452 399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "97b15e1740374ce2b8dff2e0ca20791e"
format_stamp: "Formatted at 2026-08-12 03:04:31 on dist-test-slave-zpvx"
server_key: "6104b7d18aac7b66eb7e79cd861c63bc"
server_key_iv: "3fcde542e2c7639b6c7be9f02d49eb7b"
server_key_version: "encryptionkey@0"
I20260812 03:04:31.159837 399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:31.224730 399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:31.230413 399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:31.230895 399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:31.301153 399 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:43653
I20260812 03:04:31.301278 525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:43653 every 8 connection(s)
I20260812 03:04:31.303951 399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:04:31.309269 526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:31.313585 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 399
I20260812 03:04:31.314209 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:04:31.315307 10713 external_mini_cluster.cc:1428] Setting key 4b2e9dfba086514cc15453e7ac364996
I20260812 03:04:31.333408 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e: Bootstrap starting.
I20260812 03:04:31.340538 526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e: Neither blocks nor log segments found. Creating new log.
Aug 12 03:04:31 dist-test-slave-zpvx krb5kdc[375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:31.342270 526 log.cc:824] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:31.348280 526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e: No bootstrap required, opened a new log
I20260812 03:04:31.356060 530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:31.317709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.10.118.126:43653)
0812 03:04:31.321217 (+ 3508us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:31.321246 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:31.321300 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:04:31.325319 (+ 4019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:31.325363 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:31.325719 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:31.326042 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:31.329456 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:31.331301 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:31.334102 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:31.334962 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:31.335194 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:31.342052 (+ 6858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:31.342077 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:31.342107 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:31.342213 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:31.347857 (+ 5644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:31.349268 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:31.349305 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:31.349334 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:31.349651 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:31.350319 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:31.350345 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:31.350367 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:31.351379 (+ 1012us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:31.351849 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:31.352493 (+ 644us) server_negotiation.cc:299] Negotiation successful
0812 03:04:31.353145 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3120,"thread_start_us":2713,"threads_started":1}
I20260812 03:04:31.372020 526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b15e1740374ce2b8dff2e0ca20791e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43653 } }
I20260812 03:04:31.372782 526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:31.372993 526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b15e1740374ce2b8dff2e0ca20791e, State: Initialized, Role: FOLLOWER
I20260812 03:04:31.373692 526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b15e1740374ce2b8dff2e0ca20791e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43653 } }
I20260812 03:04:31.374182 526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:04:31.374416 526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:04:31.374661 526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:31.378888 526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b15e1740374ce2b8dff2e0ca20791e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43653 } }
I20260812 03:04:31.379765 526 leader_election.cc:302] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97b15e1740374ce2b8dff2e0ca20791e; no voters:
I20260812 03:04:31.381569 526 leader_election.cc:288] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:04:31.381968 533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:31.384126 533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [term 1 LEADER]: Becoming Leader. State: Replica: 97b15e1740374ce2b8dff2e0ca20791e, State: Running, Role: LEADER
I20260812 03:04:31.384912 533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b15e1740374ce2b8dff2e0ca20791e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43653 } }
I20260812 03:04:31.385898 526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:04:31.395573 546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97b15e1740374ce2b8dff2e0ca20791e. Latest consensus state: current_term: 1 leader_uuid: "97b15e1740374ce2b8dff2e0ca20791e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b15e1740374ce2b8dff2e0ca20791e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43653 } } }
I20260812 03:04:31.396833 534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97b15e1740374ce2b8dff2e0ca20791e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b15e1740374ce2b8dff2e0ca20791e" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 43653 } } }
I20260812 03:04:31.397594 534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:31.396302 546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:31.401219 561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:04:31.413156 561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:04:31.429723 561 catalog_manager.cc:1347] Generated new cluster ID: 4790512eaa0d485cbc6d9cad29eae739
I20260812 03:04:31.430099 561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:04:31.447409 561 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:04:31.449393 561 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:04:31.462214 561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e: Generated new TSK 0
I20260812 03:04:31.463413 561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:04:31.544764 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43653
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:31.885135 574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:31.885687 574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:31.886404 574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:31.918467 574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:31.918848 574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:31.919775 574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:04:31.955454 574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:31.956733 574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:31.957785 574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:31.979087 583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:31.979980 584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:31.980475 587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:33.183388 585 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:04:33.183475 574 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:04:33.187368 574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:33.190339 574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:33.191820 574 hybrid_clock.cc:648] HybridClock initialized: now 1786503873191776 us; error 58 us; skew 500 ppm
I20260812 03:04:33.192791 574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:33 dist-test-slave-zpvx krb5kdc[375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503873, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:33.203691 574 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:04:33.209197 574 webserver.cc:460] Webserver started at http://127.10.118.65:34791/ using document root <none> and password file <none>
I20260812 03:04:33.210158 574 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:33.210407 574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:33.210878 574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:33.216161 574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "f4fc357d488948509d934ae6327f3bfe"
format_stamp: "Formatted at 2026-08-12 03:04:33 on dist-test-slave-zpvx"
server_key: "fd03c74fe33a6115b07a9b20f46d4bf3"
server_key_iv: "16ea04072dfbf4b8a5604b706a6cf9ac"
server_key_version: "encryptionkey@0"
I20260812 03:04:33.217448 574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "f4fc357d488948509d934ae6327f3bfe"
format_stamp: "Formatted at 2026-08-12 03:04:33 on dist-test-slave-zpvx"
server_key: "fd03c74fe33a6115b07a9b20f46d4bf3"
server_key_iv: "16ea04072dfbf4b8a5604b706a6cf9ac"
server_key_version: "encryptionkey@0"
I20260812 03:04:33.225302 574 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:04:33.231032 597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:33.232192 574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 03:04:33.232586 574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "f4fc357d488948509d934ae6327f3bfe"
format_stamp: "Formatted at 2026-08-12 03:04:33 on dist-test-slave-zpvx"
server_key: "fd03c74fe33a6115b07a9b20f46d4bf3"
server_key_iv: "16ea04072dfbf4b8a5604b706a6cf9ac"
server_key_version: "encryptionkey@0"
I20260812 03:04:33.232999 574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:33.285880 574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:33.292307 574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:33.292850 574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:33.295440 574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:33.299785 574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:33.300055 574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:33.300330 574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:33.300623 574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 03:04:33.445969 574 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:44367
I20260812 03:04:33.446087 783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:44367 every 8 connection(s)
I20260812 03:04:33.448796 574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:04:33.456499 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 574
I20260812 03:04:33.457193 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:04:33.458432 10713 external_mini_cluster.cc:1428] Setting key d729ed65c9104b3f9a50b10ade4761d9
Aug 12 03:04:33 dist-test-slave-zpvx krb5kdc[375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503873, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
I20260812 03:04:33.487008 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:33.453261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:52381 (local address 127.10.118.126:43653)
0812 03:04:33.454643 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:33.454671 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:33.456494 (+ 1823us) server_negotiation.cc:407] Connection header received
0812 03:04:33.462057 (+ 5563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:33.462077 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:33.462357 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:33.462527 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:33.464938 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:33.466565 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:33.468866 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:33.469980 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:33.470350 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:33.475696 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:33.475735 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:33.475752 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:33.475849 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:33.480091 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:33.481531 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:33.481566 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:33.481580 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:33.481834 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:33.482809 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:33.482831 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:33.482848 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:33.483366 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:33.483551 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:33.485041 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0812 03:04:33.485390 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":567,"threads_started":1}
I20260812 03:04:33.488416 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:33.453591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43653 (local address 127.10.118.65:52381)
0812 03:04:33.456219 (+ 2628us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:33.456294 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:33.461692 (+ 5398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:33.462866 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:33.462892 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:33.463716 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:33.464661 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:33.464709 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:33.466901 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:33.466922 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:33.468602 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:33.468656 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:33.470304 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:33.470329 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:33.470665 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:33.471595 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:33.471683 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:33.475334 (+ 3651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:33.480439 (+ 5105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:33.480476 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:33.480537 (+ 61us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:33.481311 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:33.482078 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:33.482113 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:33.482135 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:33.482531 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:33.483605 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:33.483665 (+ 60us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:33.484334 (+ 669us) client_negotiation.cc:769] Sending connection context
0812 03:04:33.484864 (+ 530us) client_negotiation.cc:240] Negotiation successful
0812 03:04:33.485644 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2188,"thread_start_us":1413,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:04:33.490604 785 heartbeater.cc:347] Connected to a master server at 127.10.118.126:43653
I20260812 03:04:33.491101 785 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:33.492251 785 heartbeater.cc:511] Master 127.10.118.126:43653 requested a full tablet report, sending...
I20260812 03:04:33.495059 461 ts_manager.cc:194] Registered new tserver with Master: f4fc357d488948509d934ae6327f3bfe (127.10.118.65:44367)
I20260812 03:04:33.497156 461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:52381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:04:33.521154 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43653
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:33.826097 791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:33.826653 791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:33.827316 791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:33.858318 791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:33.858698 791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:33.859637 791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:04:33.894320 791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:33.895495 791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:33.896607 791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:33.916210 805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:34.501959 785 heartbeater.cc:503] Master 127.10.118.126:43653 was elected leader, sending a full tablet report...
W20260812 03:04:33.916474 803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:33.916466 807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:33.917609 791 server_base.cc:1034] running on GCE node
I20260812 03:04:35.155129 791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:35.157514 791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:35.158870 791 hybrid_clock.cc:648] HybridClock initialized: now 1786503875158848 us; error 46 us; skew 500 ppm
I20260812 03:04:35.159725 791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:35 dist-test-slave-zpvx krb5kdc[375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503875, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:35.166108 791 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:04:35.171344 791 webserver.cc:460] Webserver started at http://127.10.118.66:40903/ using document root <none> and password file <none>
I20260812 03:04:35.172295 791 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:35.172590 791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:35.173050 791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:35.178408 791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "e4b1373736c244b7b6f24be2dc9dc419"
format_stamp: "Formatted at 2026-08-12 03:04:35 on dist-test-slave-zpvx"
server_key: "cec413b313cd7e81d38a8a6a2f9a6d55"
server_key_iv: "ca6d46f9f4af07cfd6118634e37f7176"
server_key_version: "encryptionkey@0"
I20260812 03:04:35.179628 791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "e4b1373736c244b7b6f24be2dc9dc419"
format_stamp: "Formatted at 2026-08-12 03:04:35 on dist-test-slave-zpvx"
server_key: "cec413b313cd7e81d38a8a6a2f9a6d55"
server_key_iv: "ca6d46f9f4af07cfd6118634e37f7176"
server_key_version: "encryptionkey@0"
I20260812 03:04:35.186858 791 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260812 03:04:35.192713 820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:35.193842 791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:04:35.194168 791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "e4b1373736c244b7b6f24be2dc9dc419"
format_stamp: "Formatted at 2026-08-12 03:04:35 on dist-test-slave-zpvx"
server_key: "cec413b313cd7e81d38a8a6a2f9a6d55"
server_key_iv: "ca6d46f9f4af07cfd6118634e37f7176"
server_key_version: "encryptionkey@0"
I20260812 03:04:35.194568 791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:35.258793 791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:35.264760 791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:35.265229 791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:35.267871 791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:35.271878 791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:35.272107 791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:35.272413 791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:35.272603 791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 03:04:35.414798 791 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:33323
I20260812 03:04:35.414927 1021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:33323 every 8 connection(s)
I20260812 03:04:35.417392 791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:04:35.420751 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 791
I20260812 03:04:35.421320 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:04:35.422771 10713 external_mini_cluster.cc:1428] Setting key e4ee399939e754abf9a0a04005b0477f
Aug 12 03:04:35 dist-test-slave-zpvx krb5kdc[375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503875, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:04:35.457271 1026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:35.422145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:43499 (local address 127.10.118.126:43653)
0812 03:04:35.423349 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:35.423387 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:35.425552 (+ 2165us) server_negotiation.cc:407] Connection header received
0812 03:04:35.430888 (+ 5336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:35.430926 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:35.431191 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:35.431395 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:35.434330 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:35.435911 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:35.438812 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:35.439949 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:35.440280 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:35.445372 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:35.445395 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:35.445415 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:35.445515 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:35.449871 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:35.451090 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:35.451119 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:35.451133 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:35.451332 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:35.452055 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:35.452074 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:35.452087 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:35.452737 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:35.453017 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:35.454432 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0812 03:04:35.454922 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":597,"threads_started":1}
I20260812 03:04:35.457923 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:35.422486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43653 (local address 127.10.118.66:43499)
0812 03:04:35.425290 (+ 2804us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:35.425389 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:35.430501 (+ 5112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:35.431665 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:35.431702 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:35.432846 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:35.433984 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:35.434043 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:35.436284 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:35.436320 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:35.438476 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:35.438534 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:35.440277 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:35.440301 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:35.440734 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:35.441690 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:35.441770 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:35.445015 (+ 3245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:35.450104 (+ 5089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:35.450142 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:35.450170 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:35.450868 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:35.451523 (+ 655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:35.451552 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:35.451567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:35.451845 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:35.452981 (+ 1136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:35.453020 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:35.453519 (+ 499us) client_negotiation.cc:769] Sending connection context
0812 03:04:35.454034 (+ 515us) client_negotiation.cc:240] Negotiation successful
0812 03:04:35.454845 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2318,"thread_start_us":1478,"threads_started":1}
I20260812 03:04:35.460196 1023 heartbeater.cc:347] Connected to a master server at 127.10.118.126:43653
I20260812 03:04:35.460640 1023 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:35.461603 1023 heartbeater.cc:511] Master 127.10.118.126:43653 requested a full tablet report, sending...
I20260812 03:04:35.463780 461 ts_manager.cc:194] Registered new tserver with Master: e4b1373736c244b7b6f24be2dc9dc419 (127.10.118.66:33323)
I20260812 03:04:35.465488 461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:43499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:04:35.487108 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43653
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:34935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:35.800761 1030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:35.801347 1030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:35.802076 1030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:35.833884 1030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:35.834280 1030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:35.835289 1030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:04:35.870883 1030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:34935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:43653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:35.872117 1030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:35.873224 1030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:35.891747 1048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:36.469437 1023 heartbeater.cc:503] Master 127.10.118.126:43653 was elected leader, sending a full tablet report...
W20260812 03:04:35.891816 1051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:35.891747 1046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:37.074199 1050 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:04:37.074409 1030 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:04:37.077963 1030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:37.080250 1030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:37.081732 1030 hybrid_clock.cc:648] HybridClock initialized: now 1786503877081701 us; error 53 us; skew 500 ppm
I20260812 03:04:37.082576 1030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:37 dist-test-slave-zpvx krb5kdc[375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503877, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:37.088457 1030 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:04:37.093777 1030 webserver.cc:460] Webserver started at http://127.10.118.67:35677/ using document root <none> and password file <none>
I20260812 03:04:37.094700 1030 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:37.094990 1030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:37.095456 1030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:37.100922 1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d"
format_stamp: "Formatted at 2026-08-12 03:04:37 on dist-test-slave-zpvx"
server_key: "0abb69a4aff76a496972112c51ca4334"
server_key_iv: "eb191af1132ee502b622a0ba60bd928a"
server_key_version: "encryptionkey@0"
I20260812 03:04:37.102141 1030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d"
format_stamp: "Formatted at 2026-08-12 03:04:37 on dist-test-slave-zpvx"
server_key: "0abb69a4aff76a496972112c51ca4334"
server_key_iv: "eb191af1132ee502b622a0ba60bd928a"
server_key_version: "encryptionkey@0"
I20260812 03:04:37.109529 1030 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:04:37.115233 1063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:37.116272 1030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 03:04:37.116654 1030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d"
format_stamp: "Formatted at 2026-08-12 03:04:37 on dist-test-slave-zpvx"
server_key: "0abb69a4aff76a496972112c51ca4334"
server_key_iv: "eb191af1132ee502b622a0ba60bd928a"
server_key_version: "encryptionkey@0"
I20260812 03:04:37.117051 1030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:37.154635 1030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:37.160439 1030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:37.160905 1030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:37.163441 1030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:37.167451 1030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:37.167685 1030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:37.167963 1030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:37.168139 1030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:37.306972 1030 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:39371
I20260812 03:04:37.307063 1240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:39371 every 8 connection(s)
I20260812 03:04:37.310534 1030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:04:37.317459 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 1030
I20260812 03:04:37.317929 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:04:37.318966 10713 external_mini_cluster.cc:1428] Setting key 2091438e85dd406343583b067be0691e
Aug 12 03:04:37 dist-test-slave-zpvx krb5kdc[375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503877, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:37.349952 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.315765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:41655 (local address 127.10.118.126:43653)
0812 03:04:37.317176 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.317203 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.321830 (+ 4627us) server_negotiation.cc:407] Connection header received
0812 03:04:37.327862 (+ 6032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.327892 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.328069 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.328243 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:37.330560 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.331512 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.333797 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.334405 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.334686 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.340416 (+ 5730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:37.340461 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:37.340478 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:37.340577 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:37.344450 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:37.345606 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:37.345627 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:37.345641 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:37.345843 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:37.346465 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:37.346484 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:37.346497 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:37.346994 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:37.347171 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.348119 (+ 948us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.348522 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":902,"thread_start_us":613,"threads_started":1}
I20260812 03:04:37.350808 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.316072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:43653 (local address 127.10.118.67:41655)
0812 03:04:37.321569 (+ 5497us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.321663 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:37.327510 (+ 5847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:37.328547 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:37.328586 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:37.329414 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:37.330275 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.330317 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.331789 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.331822 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.333540 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.333579 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.334745 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.334808 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.335245 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.336212 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:37.336343 (+ 131us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:37.340060 (+ 3717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:37.344624 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:37.344650 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:37.344671 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:37.345382 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:37.346003 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:37.346021 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:37.346038 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:37.346297 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:37.347164 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:37.347194 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:37.347592 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 03:04:37.347907 (+ 315us) client_negotiation.cc:240] Negotiation successful
0812 03:04:37.348439 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4903,"thread_start_us":4144,"threads_started":1}
I20260812 03:04:37.352725 1241 heartbeater.cc:347] Connected to a master server at 127.10.118.126:43653
I20260812 03:04:37.353161 1241 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:37.354378 1241 heartbeater.cc:511] Master 127.10.118.126:43653 requested a full tablet report, sending...
I20260812 03:04:37.356547 461 ts_manager.cc:194] Registered new tserver with Master: 8ba4cf3b6e3d4425bb5afe4749a56a1d (127.10.118.67:39371)
I20260812 03:04:37.357975 461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:41655
I20260812 03:04:37.368041 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:04:37.405207 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.383372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.10.118.126:43653)
0812 03:04:37.383771 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.383785 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.385164 (+ 1379us) server_negotiation.cc:407] Connection header received
0812 03:04:37.385621 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.385640 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.385820 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.385995 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:37.388502 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.389420 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.391318 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.391936 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.392225 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.395646 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:37.395668 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:37.395684 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:37.395795 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:37.399137 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:37.400233 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:37.400257 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:37.400271 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:37.400552 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:37.401203 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:37.401233 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:37.401253 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:37.401839 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:37.402224 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.403168 (+ 944us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.403673 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 03:04:37.411408 461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:04:37.432351 461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:04:37.484159 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.461612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:39371 (local address 127.0.0.1:57814)
0812 03:04:37.466845 (+ 5233us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.466934 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:37.467321 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:37.469194 (+ 1873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:37.469225 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:37.469338 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:37.470019 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.470070 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.473931 (+ 3861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.473964 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.478322 (+ 4358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.478384 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.480925 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.480955 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.481444 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.481682 (+ 238us) client_negotiation.cc:769] Sending connection context
0812 03:04:37.481890 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 03:04:37.482253 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2017,"mutex_wait_us":1157,"thread_start_us":1636,"threads_started":1}
I20260812 03:04:37.485884 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.461902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.10.118.67:39371)
0812 03:04:37.466905 (+ 5003us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.466940 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.467619 (+ 679us) server_negotiation.cc:407] Connection header received
0812 03:04:37.467849 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.467884 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.468244 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.468539 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:37.470434 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.472769 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.478667 (+ 5898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.480606 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.483424 (+ 2818us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.483918 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.484059 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.484398 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4482,"thread_start_us":4104,"threads_started":1}
I20260812 03:04:37.486236 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.464711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57144 (local address 127.10.118.65:44367)
0812 03:04:37.469647 (+ 4936us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.469689 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.469745 (+ 56us) server_negotiation.cc:407] Connection header received
0812 03:04:37.469952 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.469987 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.470348 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.471866 (+ 1518us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:37.472438 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.474433 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.479106 (+ 4673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.481145 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.481470 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.481965 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.484340 (+ 2375us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.484661 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4448,"thread_start_us":4079,"threads_started":1}
I20260812 03:04:37.487756 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.464083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:44367 (local address 127.0.0.1:57144)
0812 03:04:37.467950 (+ 3867us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.468179 (+ 229us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:37.468451 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:37.470647 (+ 2196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:37.470696 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:37.470804 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:37.471563 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.471612 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.474747 (+ 3135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.474776 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.478708 (+ 3932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.478765 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.482230 (+ 3465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.482262 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.482713 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.482868 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 03:04:37.483018 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 03:04:37.483322 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3423,"mutex_wait_us":565,"thread_start_us":3096,"threads_started":1}
I20260812 03:04:37.490151 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.469936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.10.118.66:33323)
0812 03:04:37.472042 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.472082 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.475884 (+ 3802us) server_negotiation.cc:407] Connection header received
0812 03:04:37.476084 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.476121 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.476502 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.476716 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:37.478494 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.480557 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.484317 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.486357 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.486636 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.487083 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.488405 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.488719 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":1327,"threads_started":1}
I20260812 03:04:37.490415 1139 tablet_service.cc:1467] Processing CreateTablet for tablet 49d93e6065f540278f6f2cac62172a84 (DEFAULT_TABLE table=test-table [id=b8a7359c85174816aba8a38d87422e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:37.492159 1139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d93e6065f540278f6f2cac62172a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:37.492463 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.471850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:33323 (local address 127.0.0.1:46560)
0812 03:04:37.475145 (+ 3295us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.475211 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:37.475474 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:37.477089 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:37.477116 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:37.477222 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:37.478125 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.478182 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.480887 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.480916 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.483955 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.484012 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.487451 (+ 3439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.487484 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.487938 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.488086 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 03:04:37.490805 (+ 2719us) client_negotiation.cc:240] Negotiation successful
0812 03:04:37.491098 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1865,"mutex_wait_us":89,"thread_start_us":1567,"threads_started":1}
I20260812 03:04:37.499516 683 tablet_service.cc:1467] Processing CreateTablet for tablet 49d93e6065f540278f6f2cac62172a84 (DEFAULT_TABLE table=test-table [id=b8a7359c85174816aba8a38d87422e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:37.499611 913 tablet_service.cc:1467] Processing CreateTablet for tablet 49d93e6065f540278f6f2cac62172a84 (DEFAULT_TABLE table=test-table [id=b8a7359c85174816aba8a38d87422e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:37.501350 683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d93e6065f540278f6f2cac62172a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:37.501600 913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d93e6065f540278f6f2cac62172a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:37.526232 1273 tablet_bootstrap.cc:492] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe: Bootstrap starting.
I20260812 03:04:37.529495 1272 tablet_bootstrap.cc:492] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Bootstrap starting.
I20260812 03:04:37.530254 1275 tablet_bootstrap.cc:492] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Bootstrap starting.
I20260812 03:04:37.533521 1273 tablet_bootstrap.cc:654] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:37.536971 1272 tablet_bootstrap.cc:654] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:37.537331 1273 log.cc:824] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:37.537480 1275 tablet_bootstrap.cc:654] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:37.539556 1272 log.cc:824] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:37.539860 1275 log.cc:824] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:37.543798 1273 tablet_bootstrap.cc:492] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe: No bootstrap required, opened a new log
I20260812 03:04:37.544263 1273 ts_tablet_manager.cc:1397] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20260812 03:04:37.545388 1272 tablet_bootstrap.cc:492] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: No bootstrap required, opened a new log
I20260812 03:04:37.545814 1272 ts_tablet_manager.cc:1397] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20260812 03:04:37.546352 1275 tablet_bootstrap.cc:492] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: No bootstrap required, opened a new log
I20260812 03:04:37.546829 1275 ts_tablet_manager.cc:1397] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260812 03:04:37.567031 1272 raft_consensus.cc:348] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.567749 1272 raft_consensus.cc:374] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:37.568033 1272 raft_consensus.cc:729] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba4cf3b6e3d4425bb5afe4749a56a1d, State: Initialized, Role: FOLLOWER
I20260812 03:04:37.568879 1272 consensus_queue.cc:260] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.572111 1241 heartbeater.cc:503] Master 127.10.118.126:43653 was elected leader, sending a full tablet report...
I20260812 03:04:37.573450 1272 ts_tablet_manager.cc:1428] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Time spent starting tablet: real 0.027s user 0.023s sys 0.004s
I20260812 03:04:37.576251 1275 raft_consensus.cc:348] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.576994 1275 raft_consensus.cc:374] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:37.577333 1275 raft_consensus.cc:729] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b1373736c244b7b6f24be2dc9dc419, State: Initialized, Role: FOLLOWER
I20260812 03:04:37.576993 1273 raft_consensus.cc:348] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.577934 1273 raft_consensus.cc:374] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:37.578253 1273 raft_consensus.cc:729] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4fc357d488948509d934ae6327f3bfe, State: Initialized, Role: FOLLOWER
I20260812 03:04:37.578302 1275 consensus_queue.cc:260] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.579144 1273 consensus_queue.cc:260] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.584540 1275 ts_tablet_manager.cc:1428] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Time spent starting tablet: real 0.037s user 0.031s sys 0.007s
I20260812 03:04:37.587144 1273 ts_tablet_manager.cc:1428] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe: Time spent starting tablet: real 0.043s user 0.034s sys 0.005s
I20260812 03:04:37.597313 1282 raft_consensus.cc:482] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:04:37.597759 1282 raft_consensus.cc:504] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.600042 1282 leader_election.cc:288] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4fc357d488948509d934ae6327f3bfe (127.10.118.65:44367), e4b1373736c244b7b6f24be2dc9dc419 (127.10.118.66:33323)
I20260812 03:04:37.611542 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.600627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:57693 (local address 127.10.118.65:44367)
0812 03:04:37.601105 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.601124 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.601190 (+ 66us) server_negotiation.cc:407] Connection header received
0812 03:04:37.601603 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.601631 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.601928 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.602131 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:37.603625 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.604987 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.607541 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.608585 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.608840 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.609154 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.609794 (+ 640us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.610170 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 03:04:37.611582 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.600532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:44367 (local address 127.10.118.67:57693)
0812 03:04:37.601028 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.601074 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:37.601315 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:37.602700 (+ 1385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:37.602736 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:37.602849 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:37.603331 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.603390 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.605257 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.605297 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.607247 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.607299 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.608858 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.608895 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.609340 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.609488 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 03:04:37.609696 (+ 208us) client_negotiation.cc:240] Negotiation successful
0812 03:04:37.609993 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 03:04:37.613574 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.601629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:49693 (local address 127.10.118.66:33323)
0812 03:04:37.602267 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.602301 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.603869 (+ 1568us) server_negotiation.cc:407] Connection header received
0812 03:04:37.604161 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.604189 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.604473 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.604622 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:37.605856 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.606675 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.609439 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.610748 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.611128 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.611389 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.611870 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.612158 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 03:04:37.613485 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d93e6065f540278f6f2cac62172a84" candidate_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4fc357d488948509d934ae6327f3bfe" is_pre_election: true
I20260812 03:04:37.614015 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.601377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:33323 (local address 127.10.118.67:49693)
0812 03:04:37.603715 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.603752 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:37.603938 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:37.604697 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:37.604715 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:37.604787 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:37.605574 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.605623 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.606939 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.606969 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.609021 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:37.609085 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.611092 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.611127 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:37.611545 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.611644 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 03:04:37.611810 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 03:04:37.612077 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843,"thread_start_us":1501,"threads_started":1}
I20260812 03:04:37.614410 715 raft_consensus.cc:2454] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba4cf3b6e3d4425bb5afe4749a56a1d in term 0.
I20260812 03:04:37.615828 1071 leader_election.cc:302] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba4cf3b6e3d4425bb5afe4749a56a1d, f4fc357d488948509d934ae6327f3bfe; no voters:
I20260812 03:04:37.615921 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d93e6065f540278f6f2cac62172a84" candidate_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b1373736c244b7b6f24be2dc9dc419" is_pre_election: true
I20260812 03:04:37.616742 1282 raft_consensus.cc:2781] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:04:37.616843 946 raft_consensus.cc:2454] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ba4cf3b6e3d4425bb5afe4749a56a1d in term 0.
I20260812 03:04:37.617039 1282 raft_consensus.cc:482] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:04:37.617342 1282 raft_consensus.cc:3037] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:37.621991 1282 raft_consensus.cc:504] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.623344 1282 leader_election.cc:288] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [CANDIDATE]: Term 1 election: Requested vote from peers f4fc357d488948509d934ae6327f3bfe (127.10.118.65:44367), e4b1373736c244b7b6f24be2dc9dc419 (127.10.118.66:33323)
I20260812 03:04:37.624157 715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d93e6065f540278f6f2cac62172a84" candidate_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4fc357d488948509d934ae6327f3bfe"
I20260812 03:04:37.624491 946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d93e6065f540278f6f2cac62172a84" candidate_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b1373736c244b7b6f24be2dc9dc419"
I20260812 03:04:37.624639 715 raft_consensus.cc:3037] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:37.624956 946 raft_consensus.cc:3037] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:37.629451 715 raft_consensus.cc:2454] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba4cf3b6e3d4425bb5afe4749a56a1d in term 1.
I20260812 03:04:37.629878 946 raft_consensus.cc:2454] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ba4cf3b6e3d4425bb5afe4749a56a1d in term 1.
I20260812 03:04:37.630661 1071 leader_election.cc:302] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ba4cf3b6e3d4425bb5afe4749a56a1d, f4fc357d488948509d934ae6327f3bfe; no voters:
I20260812 03:04:37.631399 1282 raft_consensus.cc:2781] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:37.632853 1282 raft_consensus.cc:686] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 1 LEADER]: Becoming Leader. State: Replica: 8ba4cf3b6e3d4425bb5afe4749a56a1d, State: Running, Role: LEADER
I20260812 03:04:37.633617 1282 consensus_queue.cc:237] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } }
I20260812 03:04:37.645582 461 catalog_manager.cc:5495] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ba4cf3b6e3d4425bb5afe4749a56a1d (127.10.118.67). New cstate: current_term: 1 leader_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ba4cf3b6e3d4425bb5afe4749a56a1d" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 39371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 } health_report { overall_health: UNKNOWN } } }
W20260812 03:04:37.674005 1025 tablet.cc:2369] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:37.688356 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.679680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.10.118.67:39371)
0812 03:04:37.680154 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:37.680179 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:37.680264 (+ 85us) server_negotiation.cc:407] Connection header received
0812 03:04:37.680892 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:37.680912 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:37.681136 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:37.681310 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:04:37.681933 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.682788 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.684537 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:37.685081 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:37.685293 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:37.686109 (+ 816us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:04:37.686735 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:04:37.686870 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:37.687103 (+ 233us) server_negotiation.cc:299] Negotiation successful
0812 03:04:37.687247 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260812 03:04:37.700238 946 raft_consensus.cc:1261] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 1 FOLLOWER]: Refusing update from remote peer 8ba4cf3b6e3d4425bb5afe4749a56a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:04:37.700636 715 raft_consensus.cc:1261] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe [term 1 FOLLOWER]: Refusing update from remote peer 8ba4cf3b6e3d4425bb5afe4749a56a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:04:37.701958 1291 consensus_queue.cc:1035] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b1373736c244b7b6f24be2dc9dc419" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 03:04:37.702793 1282 consensus_queue.cc:1035] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4fc357d488948509d934ae6327f3bfe" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 03:04:37.707918 786 tablet.cc:2369] T 49d93e6065f540278f6f2cac62172a84 P f4fc357d488948509d934ae6327f3bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:37.725555 1297 mvcc.cc:205] Tried to move back new op lower bound from 7317519883045195776 to 7317519882794754048. Current Snapshot: MvccSnapshot[applied={T|T < 7317519883045195776}]
I20260812 03:04:37.729104 1300 mvcc.cc:205] Tried to move back new op lower bound from 7317519883045195776 to 7317519882794754048. Current Snapshot: MvccSnapshot[applied={T|T < 7317519883045195776}]
I20260812 03:04:37.755126 458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:37.755802 458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:37.766016 458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97b15e1740374ce2b8dff2e0ca20791e: Sending DeleteTablet for 3 replicas of tablet 49d93e6065f540278f6f2cac62172a84
I20260812 03:04:37.768005 1139 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d93e6065f540278f6f2cac62172a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:37 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57814
I20260812 03:04:37.768136 683 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d93e6065f540278f6f2cac62172a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:37 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57144
I20260812 03:04:37.768906 913 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d93e6065f540278f6f2cac62172a84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:37 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:46560
I20260812 03:04:37.772029 1312 tablet_replica.cc:330] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: stopping tablet replica
I20260812 03:04:37.772614 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 574
I20260812 03:04:37.772866 1312 raft_consensus.cc:2229] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:04:37.777839 1312 raft_consensus.cc:2258] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:37.785657 1314 tablet_replica.cc:330] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: stopping tablet replica
I20260812 03:04:37.786580 1312 ts_tablet_manager.cc:1905] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:37.787110 1314 raft_consensus.cc:2229] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:04:37.788069 1314 raft_consensus.cc:2258] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:37.793434 1314 ts_tablet_manager.cc:1905] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:37.802961 1312 ts_tablet_manager.cc:1918] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:37.803401 1312 log.cc:1196] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/49d93e6065f540278f6f2cac62172a84
I20260812 03:04:37.804204 1312 ts_tablet_manager.cc:1939] T 49d93e6065f540278f6f2cac62172a84 P 8ba4cf3b6e3d4425bb5afe4749a56a1d: Deleting consensus metadata
I20260812 03:04:37.807622 431 catalog_manager.cc:4841] TS 8ba4cf3b6e3d4425bb5afe4749a56a1d (127.10.118.67:39371): tablet 49d93e6065f540278f6f2cac62172a84 (table test-table [id=b8a7359c85174816aba8a38d87422e49]) successfully deleted
I20260812 03:04:37.812592 1314 ts_tablet_manager.cc:1918] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:37.812971 1314 log.cc:1196] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/49d93e6065f540278f6f2cac62172a84
I20260812 03:04:37.813722 1314 ts_tablet_manager.cc:1939] T 49d93e6065f540278f6f2cac62172a84 P e4b1373736c244b7b6f24be2dc9dc419: Deleting consensus metadata
I20260812 03:04:37.816763 433 catalog_manager.cc:4841] TS e4b1373736c244b7b6f24be2dc9dc419 (127.10.118.66:33323): tablet 49d93e6065f540278f6f2cac62172a84 (table test-table [id=b8a7359c85174816aba8a38d87422e49]) successfully deleted
W20260812 03:04:37.850955 433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:44367 (error 108)
I20260812 03:04:37.851181 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 791
I20260812 03:04:37.873790 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.872215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:44367 (local address 127.0.0.1:57158)
0812 03:04:37.872858 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.873131 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260812 03:04:37.874424 433 catalog_manager.cc:4584] TS f4fc357d488948509d934ae6327f3bfe (127.10.118.65:44367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49d93e6065f540278f6f2cac62172a84: Network error: Client connection negotiation failed: client connection to 127.10.118.65:44367: connect: Connection refused (error 111)
I20260812 03:04:37.895934 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 1030
I20260812 03:04:37.930323 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.929233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:44367 (local address 127.0.0.1:57174)
0812 03:04:37.929724 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.929892 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260812 03:04:37.932653 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:37.931614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:44367 (local address 127.0.0.1:57184)
0812 03:04:37.932060 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:37.932209 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:44367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 03:04:37.936987 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 399
2026-08-12T03:04:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (10068 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T03:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:04:38Z Disabled control of system clock
I20260812 03:04:38.036821 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44655
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:44655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 03:04:38.347299 1322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:38.347856 1322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:38.348325 1322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:38.378810 1322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:04:38.379106 1322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:38.379273 1322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:38.379465 1322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:04:38.379652 1322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:04:38.414147 1322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:44655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:38.415314 1322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:38.416957 1322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:38.433668 1338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:38.433679 1339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:38.435528 1322 server_base.cc:1034] running on GCE node
W20260812 03:04:38.434309 1342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:39.634251 1322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:39.636952 1322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:39.638309 1322 hybrid_clock.cc:648] HybridClock initialized: now 1786503879638286 us; error 47 us; skew 500 ppm
I20260812 03:04:39.639199 1322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:04:39.651530 1322 webserver.cc:460] Webserver started at http://127.10.118.126:38239/ using document root <none> and password file <none>
I20260812 03:04:39.652575 1322 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:39.652813 1322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:39.653311 1322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:39.658715 1322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "0b25b3fdcaf84573834fceb443e04e4a"
format_stamp: "Formatted at 2026-08-12 03:04:39 on dist-test-slave-zpvx"
server_key: "2a321b245cfd0064599405f74431d40d"
server_key_iv: "a31555c28b1fa6d8634bc4070b45375d"
server_key_version: "encryptionkey@0"
I20260812 03:04:39.659899 1322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "0b25b3fdcaf84573834fceb443e04e4a"
format_stamp: "Formatted at 2026-08-12 03:04:39 on dist-test-slave-zpvx"
server_key: "2a321b245cfd0064599405f74431d40d"
server_key_iv: "a31555c28b1fa6d8634bc4070b45375d"
server_key_version: "encryptionkey@0"
I20260812 03:04:39.667339 1322 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 03:04:39.673098 1353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:39.674139 1322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 03:04:39.674484 1322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "0b25b3fdcaf84573834fceb443e04e4a"
format_stamp: "Formatted at 2026-08-12 03:04:39 on dist-test-slave-zpvx"
server_key: "2a321b245cfd0064599405f74431d40d"
server_key_iv: "a31555c28b1fa6d8634bc4070b45375d"
server_key_version: "encryptionkey@0"
I20260812 03:04:39.674854 1322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:39.722038 1322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:39.723532 1322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:39.724064 1322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:39.801023 1322 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:44655
I20260812 03:04:39.801203 1443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:44655 every 8 connection(s)
I20260812 03:04:39.804967 1322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:04:39.812259 1446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:39.813620 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 1322
I20260812 03:04:39.814117 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:04:39.815177 10713 external_mini_cluster.cc:1428] Setting key 0018310e76d72a4e73be2fdd6e1bfe27
I20260812 03:04:39.849014 1446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a: Bootstrap starting.
I20260812 03:04:39.850353 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:39.817822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.10.118.126:44655)
0812 03:04:39.826183 (+ 8361us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:39.826227 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:39.826313 (+ 86us) server_negotiation.cc:407] Connection header received
0812 03:04:39.832975 (+ 6662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:39.833036 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:39.833661 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:39.834171 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:39.836603 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:39.838272 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:39.841246 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:39.842287 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:39.842631 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:39.844715 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:39.844748 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:39.844786 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:04:39.845835 (+ 1049us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:39.845966 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:04:39.846025 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:39.846178 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:39.847474 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0812 03:04:39.847844 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7560,"thread_start_us":7017,"threads_started":1}
I20260812 03:04:39.855927 1446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:39.858206 1446 log.cc:824] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:39.864876 1446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a: No bootstrap required, opened a new log
I20260812 03:04:39.886250 1446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b25b3fdcaf84573834fceb443e04e4a" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44655 } }
I20260812 03:04:39.886982 1446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:39.887259 1446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b25b3fdcaf84573834fceb443e04e4a, State: Initialized, Role: FOLLOWER
I20260812 03:04:39.888094 1446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b25b3fdcaf84573834fceb443e04e4a" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44655 } }
I20260812 03:04:39.888828 1446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:04:39.889178 1446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:04:39.889561 1446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:39.893671 1446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b25b3fdcaf84573834fceb443e04e4a" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44655 } }
I20260812 03:04:39.894515 1446 leader_election.cc:302] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b25b3fdcaf84573834fceb443e04e4a; no voters:
I20260812 03:04:39.896345 1446 leader_election.cc:288] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:04:39.896807 1456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:39.900220 1456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [term 1 LEADER]: Becoming Leader. State: Replica: 0b25b3fdcaf84573834fceb443e04e4a, State: Running, Role: LEADER
I20260812 03:04:39.901001 1456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b25b3fdcaf84573834fceb443e04e4a" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44655 } }
I20260812 03:04:39.901225 1446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:04:39.912127 1457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b25b3fdcaf84573834fceb443e04e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b25b3fdcaf84573834fceb443e04e4a" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44655 } } }
I20260812 03:04:39.914316 1457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:39.913632 1458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b25b3fdcaf84573834fceb443e04e4a. Latest consensus state: current_term: 1 leader_uuid: "0b25b3fdcaf84573834fceb443e04e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b25b3fdcaf84573834fceb443e04e4a" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44655 } } }
I20260812 03:04:39.914752 1458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:39.917925 1480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:04:39.929593 1480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:04:39.945878 1480 catalog_manager.cc:1347] Generated new cluster ID: 3b95b98b63364550b62e44e1d3f0ca21
I20260812 03:04:39.946225 1480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:04:39.984639 1480 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:04:39.986730 1480 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:04:40.003798 1480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a: Generated new TSK 0
I20260812 03:04:40.005017 1480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:04:40.027568 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44655
--builtin_ntp_servers=127.10.118.84:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 03:04:40.342535 1495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:40.343099 1495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:40.343667 1495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:40.375195 1495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:40.375494 1495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:40.376354 1495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:04:40.411096 1495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:40.412460 1495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:40.414140 1495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:40.436041 1511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:40.436801 1508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:40.438838 1495 server_base.cc:1034] running on GCE node
W20260812 03:04:40.437107 1515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:41.645684 1495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:41.647994 1495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:41.649348 1495 hybrid_clock.cc:648] HybridClock initialized: now 1786503881649305 us; error 60 us; skew 500 ppm
I20260812 03:04:41.650179 1495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:04:41.656636 1495 webserver.cc:460] Webserver started at http://127.10.118.65:35695/ using document root <none> and password file <none>
I20260812 03:04:41.657585 1495 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:41.657819 1495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:41.658289 1495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:41.663596 1495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "16f66938a6e5438ba9a8929c66b21fef"
format_stamp: "Formatted at 2026-08-12 03:04:41 on dist-test-slave-zpvx"
server_key: "5daa8730fb5903a65c73b9a3ca198089"
server_key_iv: "8eeb042afeac1fca468a8b1895e2c6b5"
server_key_version: "encryptionkey@0"
I20260812 03:04:41.664861 1495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "16f66938a6e5438ba9a8929c66b21fef"
format_stamp: "Formatted at 2026-08-12 03:04:41 on dist-test-slave-zpvx"
server_key: "5daa8730fb5903a65c73b9a3ca198089"
server_key_iv: "8eeb042afeac1fca468a8b1895e2c6b5"
server_key_version: "encryptionkey@0"
I20260812 03:04:41.672137 1495 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:04:41.677742 1526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:41.678834 1495 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:04:41.679245 1495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "16f66938a6e5438ba9a8929c66b21fef"
format_stamp: "Formatted at 2026-08-12 03:04:41 on dist-test-slave-zpvx"
server_key: "5daa8730fb5903a65c73b9a3ca198089"
server_key_iv: "8eeb042afeac1fca468a8b1895e2c6b5"
server_key_version: "encryptionkey@0"
I20260812 03:04:41.679620 1495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:41.718173 1495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:41.719792 1495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:41.720257 1495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:41.722798 1495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:41.726804 1495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:41.727041 1495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:41.727295 1495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:41.727468 1495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:41.870358 1495 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:43309
I20260812 03:04:41.870479 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:43309 every 8 connection(s)
I20260812 03:04:41.873024 1495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:04:41.874033 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 1495
I20260812 03:04:41.874588 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:04:41.876227 10713 external_mini_cluster.cc:1428] Setting key 7780ad1ad173298c76599389e033aaa3
I20260812 03:04:41.882365 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44655
--builtin_ntp_servers=127.10.118.84:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 03:04:41.898383 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:41.878535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:52547 (local address 127.10.118.126:44655)
0812 03:04:41.879832 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:41.879855 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:41.882410 (+ 2555us) server_negotiation.cc:407] Connection header received
0812 03:04:41.887295 (+ 4885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:41.887333 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:41.887507 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:41.887685 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:41.889440 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:41.890400 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:41.892549 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:41.893158 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:41.893454 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:41.895548 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:41.895575 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:41.895591 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:04:41.895635 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:41.895702 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:04:41.895737 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:41.895883 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:41.896229 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 03:04:41.896469 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":562,"threads_started":1}
I20260812 03:04:41.899227 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:41.878831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:44655 (local address 127.10.118.65:52547)
0812 03:04:41.882190 (+ 3359us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:41.882256 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:41.886940 (+ 4684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:41.887973 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:41.888006 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:41.888164 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:41.889152 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:41.889197 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:41.890667 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:41.890687 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:41.892262 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:41.892302 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:41.893512 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:41.893550 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:41.894041 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:41.894975 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:04:41.895006 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:41.895091 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:04:41.895220 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:41.895903 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:41.895950 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:41.895981 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 03:04:41.896158 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 03:04:41.896607 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2699,"thread_start_us":1808,"threads_started":1}
I20260812 03:04:41.901409 1718 heartbeater.cc:347] Connected to a master server at 127.10.118.126:44655
I20260812 03:04:41.901834 1718 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:41.902902 1718 heartbeater.cc:511] Master 127.10.118.126:44655 requested a full tablet report, sending...
I20260812 03:04:41.905546 1382 ts_manager.cc:194] Registered new tserver with Master: 16f66938a6e5438ba9a8929c66b21fef (127.10.118.65:43309)
I20260812 03:04:41.907593 1382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.65:52547
W20260812 03:04:42.189370 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:42.189929 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:42.190522 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:42.222368 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:42.222638 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:42.223520 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:04:42.257500 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:42.258710 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:42.260351 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:42.280025 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:42.911521 1718 heartbeater.cc:503] Master 127.10.118.126:44655 was elected leader, sending a full tablet report...
W20260812 03:04:42.280054 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:42.280689 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:42.286121 1722 server_base.cc:1034] running on GCE node
I20260812 03:04:43.474990 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:43.477337 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:43.478703 1722 hybrid_clock.cc:648] HybridClock initialized: now 1786503883478671 us; error 52 us; skew 500 ppm
I20260812 03:04:43.479485 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:04:43.486455 1722 webserver.cc:460] Webserver started at http://127.10.118.66:39667/ using document root <none> and password file <none>
I20260812 03:04:43.487396 1722 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:43.487582 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:43.488066 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:43.493717 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "69979f2d34b849758aab62eae84869d2"
format_stamp: "Formatted at 2026-08-12 03:04:43 on dist-test-slave-zpvx"
server_key: "d034fee8c2dd8aea495b504f487302a7"
server_key_iv: "c30edc60aeb2c27a6913081b6ff58d6e"
server_key_version: "encryptionkey@0"
I20260812 03:04:43.494936 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "69979f2d34b849758aab62eae84869d2"
format_stamp: "Formatted at 2026-08-12 03:04:43 on dist-test-slave-zpvx"
server_key: "d034fee8c2dd8aea495b504f487302a7"
server_key_iv: "c30edc60aeb2c27a6913081b6ff58d6e"
server_key_version: "encryptionkey@0"
I20260812 03:04:43.502885 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260812 03:04:43.508667 1744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:43.509754 1722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 03:04:43.510110 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "69979f2d34b849758aab62eae84869d2"
format_stamp: "Formatted at 2026-08-12 03:04:43 on dist-test-slave-zpvx"
server_key: "d034fee8c2dd8aea495b504f487302a7"
server_key_iv: "c30edc60aeb2c27a6913081b6ff58d6e"
server_key_version: "encryptionkey@0"
I20260812 03:04:43.510468 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:43.557310 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:43.558818 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:43.559307 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:43.561787 1722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:43.565932 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:43.566128 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:43.566323 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:43.566452 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:43.704329 1722 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:35983
I20260812 03:04:43.704487 1927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:35983 every 8 connection(s)
I20260812 03:04:43.706908 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:04:43.713168 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 1722
I20260812 03:04:43.714040 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:04:43.715693 10713 external_mini_cluster.cc:1428] Setting key fa1ed4c2e8f7a0c063717a656259288d
I20260812 03:04:43.722900 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44655
--builtin_ntp_servers=127.10.118.84:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 03:04:43.734525 1933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:43.711472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:44655 (local address 127.10.118.66:36637)
0812 03:04:43.714594 (+ 3122us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:43.714693 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:43.721165 (+ 6472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:43.722161 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:43.722200 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:43.722397 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:43.723741 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:43.723810 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:43.725281 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:43.725320 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:43.728014 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:43.728075 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:43.729313 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:43.729357 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:43.730013 (+ 656us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:43.731059 (+ 1046us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:04:43.731086 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:43.731182 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:04:43.731361 (+ 179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:43.732031 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:43.732086 (+ 55us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:43.732124 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 03:04:43.732349 (+ 225us) client_negotiation.cc:240] Negotiation successful
0812 03:04:43.732873 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2384,"thread_start_us":1577,"threads_started":1}
I20260812 03:04:43.734726 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:43.711531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:36637 (local address 127.10.118.126:44655)
0812 03:04:43.712908 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:43.712950 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:43.714861 (+ 1911us) server_negotiation.cc:407] Connection header received
0812 03:04:43.721516 (+ 6655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:43.721537 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:43.721699 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:43.721834 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:43.724045 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:43.725012 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:43.728291 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:43.728935 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:43.729199 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:43.731675 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:43.731698 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:43.731716 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:04:43.731762 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:43.731829 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:04:43.731864 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:43.731972 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:43.732446 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 03:04:43.732704 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":528,"threads_started":1}
I20260812 03:04:43.736675 1928 heartbeater.cc:347] Connected to a master server at 127.10.118.126:44655
I20260812 03:04:43.737165 1928 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:43.738459 1928 heartbeater.cc:511] Master 127.10.118.126:44655 requested a full tablet report, sending...
I20260812 03:04:43.741197 1382 ts_manager.cc:194] Registered new tserver with Master: 69979f2d34b849758aab62eae84869d2 (127.10.118.66:35983)
I20260812 03:04:43.742781 1382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.66:36637
W20260812 03:04:44.038524 1934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:44.039004 1934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:44.039510 1934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:44.072656 1934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:44.072892 1934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:44.073679 1934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:04:44.108907 1934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:44655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:44.110142 1934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:44.111701 1934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:44.129451 1949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:44.747848 1928 heartbeater.cc:503] Master 127.10.118.126:44655 was elected leader, sending a full tablet report...
W20260812 03:04:44.129439 1948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:44.130328 1952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:45.316097 1950 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:04:45.316226 1934 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:04:45.320187 1934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:45.322549 1934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:45.323949 1934 hybrid_clock.cc:648] HybridClock initialized: now 1786503885323915 us; error 43 us; skew 500 ppm
I20260812 03:04:45.324856 1934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:04:45.335932 1934 webserver.cc:460] Webserver started at http://127.10.118.67:33421/ using document root <none> and password file <none>
I20260812 03:04:45.336966 1934 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:45.337204 1934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:45.337666 1934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:45.342933 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "3ee36e228a744157be9f55ea714b3109"
format_stamp: "Formatted at 2026-08-12 03:04:45 on dist-test-slave-zpvx"
server_key: "e5066be708bd45fd22a454a50030f3ba"
server_key_iv: "f6ff2bdf1d744c9db9ccf2620eb75adf"
server_key_version: "encryptionkey@0"
I20260812 03:04:45.344097 1934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "3ee36e228a744157be9f55ea714b3109"
format_stamp: "Formatted at 2026-08-12 03:04:45 on dist-test-slave-zpvx"
server_key: "e5066be708bd45fd22a454a50030f3ba"
server_key_iv: "f6ff2bdf1d744c9db9ccf2620eb75adf"
server_key_version: "encryptionkey@0"
I20260812 03:04:45.351238 1934 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 03:04:45.357065 1962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:45.358173 1934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:04:45.358543 1934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "3ee36e228a744157be9f55ea714b3109"
format_stamp: "Formatted at 2026-08-12 03:04:45 on dist-test-slave-zpvx"
server_key: "e5066be708bd45fd22a454a50030f3ba"
server_key_iv: "f6ff2bdf1d744c9db9ccf2620eb75adf"
server_key_version: "encryptionkey@0"
I20260812 03:04:45.359050 1934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:45.395579 1934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:45.397058 1934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:45.397521 1934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:45.400182 1934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:45.404491 1934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:45.404711 1934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:45.404978 1934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:45.405148 1934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:45.543747 1934 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:41137
I20260812 03:04:45.543860 2159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:41137 every 8 connection(s)
I20260812 03:04:45.546394 1934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:04:45.554695 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 1934
I20260812 03:04:45.555133 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:04:45.556151 10713 external_mini_cluster.cc:1428] Setting key cf2c41cd22976fd7088e7e8f2a1ad990
I20260812 03:04:45.570051 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.550652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:37455 (local address 127.10.118.126:44655)
0812 03:04:45.551737 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.551756 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.554461 (+ 2705us) server_negotiation.cc:407] Connection header received
0812 03:04:45.560186 (+ 5725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.560206 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.560418 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.560569 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:45.562295 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.563286 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.565406 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.566086 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.566290 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.568024 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:45.568045 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:45.568062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:04:45.568110 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:45.568177 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:04:45.568211 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:45.568320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.568675 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.568830 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":496,"threads_started":1}
I20260812 03:04:45.570641 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.550978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:44655 (local address 127.10.118.67:37455)
0812 03:04:45.554231 (+ 3253us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:45.554300 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:45.559835 (+ 5535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:45.560910 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:45.560942 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:45.561118 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:45.562026 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.562071 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.563524 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.563545 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.565185 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.565225 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.566369 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.566393 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.566751 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.567537 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:04:45.567553 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:45.567616 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:04:45.567735 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:45.568357 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:45.568431 (+ 74us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:45.568451 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 03:04:45.568594 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 03:04:45.568902 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2589,"thread_start_us":1735,"threads_started":1}
I20260812 03:04:45.572679 2160 heartbeater.cc:347] Connected to a master server at 127.10.118.126:44655
I20260812 03:04:45.573081 2160 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:45.573992 2160 heartbeater.cc:511] Master 127.10.118.126:44655 requested a full tablet report, sending...
I20260812 03:04:45.576018 1382 ts_manager.cc:194] Registered new tserver with Master: 3ee36e228a744157be9f55ea714b3109 (127.10.118.67:41137)
I20260812 03:04:45.577478 1382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.67:37455
I20260812 03:04:45.589273 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:04:45.616600 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.605496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.10.118.126:44655)
0812 03:04:45.605897 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.605916 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.607006 (+ 1090us) server_negotiation.cc:407] Connection header received
0812 03:04:45.607461 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.607481 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.607654 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.607820 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:45.609002 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.609923 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.611940 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.612572 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.612858 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.614463 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:45.614484 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:45.614502 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:04:45.614544 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:45.614609 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:04:45.614644 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:45.614788 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.615077 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.615330 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 03:04:45.622845 1382 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:04:45.643124 1382 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:04:45.694685 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.670676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.10.118.66:35983)
0812 03:04:45.679561 (+ 8885us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.679608 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.679684 (+ 76us) server_negotiation.cc:407] Connection header received
0812 03:04:45.679931 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.679971 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.680470 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.682545 (+ 2075us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:45.683171 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.685309 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.689153 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.691316 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.691629 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.692161 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.692810 (+ 649us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.693131 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8195,"thread_start_us":7731,"threads_started":1}
I20260812 03:04:45.695181 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.670230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:35983 (local address 127.0.0.1:49846)
0812 03:04:45.674097 (+ 3867us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:45.674205 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:45.674631 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:45.680908 (+ 6277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:45.680959 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:45.681124 (+ 165us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:45.682201 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.682259 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.685626 (+ 3367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.685669 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.688532 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.688570 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.691678 (+ 3108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.691710 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.692233 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.692509 (+ 276us) client_negotiation.cc:769] Sending connection context
0812 03:04:45.692723 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 03:04:45.693142 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3199,"mutex_wait_us":129,"thread_start_us":4150,"threads_started":1}
I20260812 03:04:45.700295 1822 tablet_service.cc:1467] Processing CreateTablet for tablet 75d2028e0d9b44649bc95db345286ea1 (DEFAULT_TABLE table=test-table [id=355f63fd8032476ab551fcbea1a43abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:45.702168 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.673261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.10.118.65:43309)
0812 03:04:45.686676 (+ 13415us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.686704 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.686756 (+ 52us) server_negotiation.cc:407] Connection header received
0812 03:04:45.686944 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.686972 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.687333 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.687572 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:45.690252 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.692201 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.696673 (+ 4472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.698665 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.698965 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.699601 (+ 636us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.699911 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.700319 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12928,"thread_start_us":12558,"threads_started":1}
I20260812 03:04:45.702171 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.673033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43309 (local address 127.0.0.1:60722)
0812 03:04:45.683459 (+ 10426us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:45.683535 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:45.683860 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:45.687816 (+ 3956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:45.687845 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:45.687953 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:45.689858 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.689912 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.692487 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.692521 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.695705 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.695770 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.698983 (+ 3213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.699021 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.699502 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.699646 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 03:04:45.699880 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 03:04:45.700240 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9267,"mutex_wait_us":205,"thread_start_us":8909,"threads_started":1}
I20260812 03:04:45.702937 1822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d2028e0d9b44649bc95db345286ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:45.705024 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.679952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:41137 (local address 127.0.0.1:55852)
0812 03:04:45.685851 (+ 5899us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:45.685910 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:45.686157 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:45.687220 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:45.687255 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:45.687348 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:45.688205 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.688259 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.691107 (+ 2848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.691138 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.694762 (+ 3624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.694895 (+ 133us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.698321 (+ 3426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.698347 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.698620 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.698851 (+ 231us) client_negotiation.cc:769] Sending connection context
0812 03:04:45.699049 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 03:04:45.700207 (+ 1158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5429,"mutex_wait_us":1085,"thread_start_us":5133,"threads_started":1}
I20260812 03:04:45.707113 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.676245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.10.118.67:41137)
0812 03:04:45.680216 (+ 3971us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.680255 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.686308 (+ 6053us) server_negotiation.cc:407] Connection header received
0812 03:04:45.686523 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.686558 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.686925 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.687162 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:45.688598 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.690840 (+ 2242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.695205 (+ 4365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.697256 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.697540 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.698064 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.699409 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.699700 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3427,"thread_start_us":2935,"threads_started":1}
I20260812 03:04:45.711030 1604 tablet_service.cc:1467] Processing CreateTablet for tablet 75d2028e0d9b44649bc95db345286ea1 (DEFAULT_TABLE table=test-table [id=355f63fd8032476ab551fcbea1a43abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:45.711408 2044 tablet_service.cc:1467] Processing CreateTablet for tablet 75d2028e0d9b44649bc95db345286ea1 (DEFAULT_TABLE table=test-table [id=355f63fd8032476ab551fcbea1a43abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:45.713178 1604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d2028e0d9b44649bc95db345286ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:45.713507 2044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d2028e0d9b44649bc95db345286ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:45.735543 2194 tablet_bootstrap.cc:492] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Bootstrap starting.
I20260812 03:04:45.742908 2196 tablet_bootstrap.cc:492] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Bootstrap starting.
I20260812 03:04:45.742933 2194 tablet_bootstrap.cc:654] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:45.745333 2194 log.cc:824] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:45.749853 2196 tablet_bootstrap.cc:654] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:45.751595 2194 tablet_bootstrap.cc:492] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: No bootstrap required, opened a new log
I20260812 03:04:45.752032 2194 ts_tablet_manager.cc:1397] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20260812 03:04:45.752560 2196 log.cc:824] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:45.766583 2199 tablet_bootstrap.cc:492] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef: Bootstrap starting.
I20260812 03:04:45.769097 2196 tablet_bootstrap.cc:492] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: No bootstrap required, opened a new log
I20260812 03:04:45.769586 2196 ts_tablet_manager.cc:1397] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.006s
I20260812 03:04:45.771869 2199 tablet_bootstrap.cc:654] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:45.773663 2199 log.cc:824] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:45.779078 2199 tablet_bootstrap.cc:492] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef: No bootstrap required, opened a new log
I20260812 03:04:45.779558 2199 ts_tablet_manager.cc:1397] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260812 03:04:45.783300 2194 raft_consensus.cc:348] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.784195 2194 raft_consensus.cc:374] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:45.784538 2194 raft_consensus.cc:729] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69979f2d34b849758aab62eae84869d2, State: Initialized, Role: FOLLOWER
I20260812 03:04:45.785540 2194 consensus_queue.cc:260] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.789719 2194 ts_tablet_manager.cc:1428] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Time spent starting tablet: real 0.037s user 0.034s sys 0.004s
I20260812 03:04:45.803136 2196 raft_consensus.cc:348] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.803823 2196 raft_consensus.cc:374] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:45.804072 2196 raft_consensus.cc:729] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee36e228a744157be9f55ea714b3109, State: Initialized, Role: FOLLOWER
I20260812 03:04:45.804903 2196 consensus_queue.cc:260] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.806913 2199 raft_consensus.cc:348] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.807761 2199 raft_consensus.cc:374] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:45.808033 2199 raft_consensus.cc:729] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f66938a6e5438ba9a8929c66b21fef, State: Initialized, Role: FOLLOWER
I20260812 03:04:45.808938 2196 ts_tablet_manager.cc:1428] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Time spent starting tablet: real 0.039s user 0.028s sys 0.009s
I20260812 03:04:45.808951 2199 consensus_queue.cc:260] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.808208 2160 heartbeater.cc:503] Master 127.10.118.126:44655 was elected leader, sending a full tablet report...
I20260812 03:04:45.816278 2199 ts_tablet_manager.cc:1428] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef: Time spent starting tablet: real 0.036s user 0.029s sys 0.005s
I20260812 03:04:45.826913 2203 raft_consensus.cc:482] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:04:45.827425 2203 raft_consensus.cc:504] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.832850 2203 leader_election.cc:288] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ee36e228a744157be9f55ea714b3109 (127.10.118.67:41137), 16f66938a6e5438ba9a8929c66b21fef (127.10.118.65:43309)
I20260812 03:04:45.846438 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.832624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:41845 (local address 127.10.118.65:43309)
0812 03:04:45.833146 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.833176 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.835205 (+ 2029us) server_negotiation.cc:407] Connection header received
0812 03:04:45.835631 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.835651 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.835838 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.835986 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:45.837512 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.838825 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.841969 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.843326 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.843549 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.843703 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.844777 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.845045 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 03:04:45.847129 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.831194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:44759 (local address 127.10.118.67:41137)
0812 03:04:45.831715 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.831743 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.834054 (+ 2311us) server_negotiation.cc:407] Connection header received
0812 03:04:45.834650 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.834680 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.834942 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.835085 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:45.836797 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.837933 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.841208 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.842746 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.843097 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.843333 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.845387 (+ 2054us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.845664 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 03:04:45.847440 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.831401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43309 (local address 127.10.118.66:41845)
0812 03:04:45.835025 (+ 3624us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:45.835076 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:45.835319 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:45.836041 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:45.836070 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:45.836168 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:45.837210 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.837266 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.839073 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.839115 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.841626 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.841702 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.843941 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.843964 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.844341 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.844518 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 03:04:45.845282 (+ 764us) client_negotiation.cc:240] Negotiation successful
0812 03:04:45.845569 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3160,"mutex_wait_us":129,"thread_start_us":2756,"threads_started":1}
I20260812 03:04:45.849401 2208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.831018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:41137 (local address 127.10.118.66:44759)
0812 03:04:45.833859 (+ 2841us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:45.833932 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:45.834310 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:45.835454 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:45.835481 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:45.835591 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:45.836484 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.836536 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.838162 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.838195 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.840904 (+ 2709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:45.840954 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.844353 (+ 3399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.844603 (+ 250us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:45.845014 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.845145 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 03:04:45.847171 (+ 2026us) client_negotiation.cc:240] Negotiation successful
0812 03:04:45.847913 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2112,"mutex_wait_us":217,"thread_start_us":1799,"threads_started":1}
I20260812 03:04:45.849262 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2028e0d9b44649bc95db345286ea1" candidate_uuid: "69979f2d34b849758aab62eae84869d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f66938a6e5438ba9a8929c66b21fef" is_pre_election: true
I20260812 03:04:45.850235 1638 raft_consensus.cc:2454] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69979f2d34b849758aab62eae84869d2 in term 0.
I20260812 03:04:45.851023 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2028e0d9b44649bc95db345286ea1" candidate_uuid: "69979f2d34b849758aab62eae84869d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee36e228a744157be9f55ea714b3109" is_pre_election: true
I20260812 03:04:45.851588 1749 leader_election.cc:302] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f66938a6e5438ba9a8929c66b21fef, 69979f2d34b849758aab62eae84869d2; no voters:
I20260812 03:04:45.851847 2084 raft_consensus.cc:2454] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69979f2d34b849758aab62eae84869d2 in term 0.
I20260812 03:04:45.852545 2203 raft_consensus.cc:2781] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:04:45.852859 2203 raft_consensus.cc:482] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:04:45.853142 2203 raft_consensus.cc:3037] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:45.859707 2203 raft_consensus.cc:504] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
I20260812 03:04:45.861603 2203 leader_election.cc:288] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [CANDIDATE]: Term 1 election: Requested vote from peers 3ee36e228a744157be9f55ea714b3109 (127.10.118.67:41137), 16f66938a6e5438ba9a8929c66b21fef (127.10.118.65:43309)
I20260812 03:04:45.862112 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2028e0d9b44649bc95db345286ea1" candidate_uuid: "69979f2d34b849758aab62eae84869d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ee36e228a744157be9f55ea714b3109"
I20260812 03:04:45.862340 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d2028e0d9b44649bc95db345286ea1" candidate_uuid: "69979f2d34b849758aab62eae84869d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f66938a6e5438ba9a8929c66b21fef"
I20260812 03:04:45.862597 2084 raft_consensus.cc:3037] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:45.862807 1638 raft_consensus.cc:3037] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:45.867331 2084 raft_consensus.cc:2454] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69979f2d34b849758aab62eae84869d2 in term 1.
I20260812 03:04:45.867414 1638 raft_consensus.cc:2454] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69979f2d34b849758aab62eae84869d2 in term 1.
I20260812 03:04:45.868440 1748 leader_election.cc:302] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ee36e228a744157be9f55ea714b3109, 69979f2d34b849758aab62eae84869d2; no voters:
I20260812 03:04:45.869218 2203 raft_consensus.cc:2781] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:45.871088 2203 raft_consensus.cc:686] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 1 LEADER]: Becoming Leader. State: Replica: 69979f2d34b849758aab62eae84869d2, State: Running, Role: LEADER
I20260812 03:04:45.872051 2203 consensus_queue.cc:237] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } }
W20260812 03:04:45.882473 1719 tablet.cc:2369] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:45.882928 1381 catalog_manager.cc:5495] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69979f2d34b849758aab62eae84869d2 (127.10.118.66). New cstate: current_term: 1 leader_uuid: "69979f2d34b849758aab62eae84869d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69979f2d34b849758aab62eae84869d2" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 35983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260812 03:04:45.950134 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:45.942167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.10.118.66:35983)
0812 03:04:45.942599 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:45.942618 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:45.942654 (+ 36us) server_negotiation.cc:407] Connection header received
0812 03:04:45.943000 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:45.943017 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:45.943259 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:45.943403 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:04:45.944033 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.944923 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.946608 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:45.947164 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:45.947352 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:45.948046 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:04:45.948547 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:04:45.948683 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:45.948889 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 03:04:45.949017 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 03:04:45.962625 1638 raft_consensus.cc:1261] T 75d2028e0d9b44649bc95db345286ea1 P 16f66938a6e5438ba9a8929c66b21fef [term 1 FOLLOWER]: Refusing update from remote peer 69979f2d34b849758aab62eae84869d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:04:45.962785 2084 raft_consensus.cc:1261] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 1 FOLLOWER]: Refusing update from remote peer 69979f2d34b849758aab62eae84869d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 03:04:45.963567 1930 tablet.cc:2369] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:45.964582 2211 consensus_queue.cc:1035] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee36e228a744157be9f55ea714b3109" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:04:45.965240 2203 consensus_queue.cc:1035] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f66938a6e5438ba9a8929c66b21fef" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:04:45.986269 2217 mvcc.cc:205] Tried to move back new op lower bound from 7317519916886761472 to 7317519916541784064. Current Snapshot: MvccSnapshot[applied={T|T < 7317519916886761472}]
I20260812 03:04:45.990314 2218 mvcc.cc:205] Tried to move back new op lower bound from 7317519916886761472 to 7317519916541784064. Current Snapshot: MvccSnapshot[applied={T|T < 7317519916886761472}]
I20260812 03:04:46.019625 1382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:46.020116 1382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:46.028522 1382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b25b3fdcaf84573834fceb443e04e4a: Sending DeleteTablet for 3 replicas of tablet 75d2028e0d9b44649bc95db345286ea1
I20260812 03:04:46.030685 1822 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d2028e0d9b44649bc95db345286ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:46 UTC) from {username='slave'} at 127.0.0.1:49846
I20260812 03:04:46.030880 1604 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d2028e0d9b44649bc95db345286ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:46 UTC) from {username='slave'} at 127.0.0.1:60722
I20260812 03:04:46.031281 2044 tablet_service.cc:1514] Processing DeleteTablet for tablet 75d2028e0d9b44649bc95db345286ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:46 UTC) from {username='slave'} at 127.0.0.1:55852
I20260812 03:04:46.035024 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 1495
I20260812 03:04:46.062505 2237 tablet_replica.cc:330] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: stopping tablet replica
I20260812 03:04:46.063588 2237 raft_consensus.cc:2229] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:04:46.064486 2237 raft_consensus.cc:2258] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:46.066079 2234 tablet_replica.cc:330] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: stopping tablet replica
I20260812 03:04:46.067649 2234 raft_consensus.cc:2229] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:04:46.068252 2237 ts_tablet_manager.cc:1905] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:46.069286 2234 raft_consensus.cc:2258] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:46.073377 2234 ts_tablet_manager.cc:1905] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:46.083231 2237 ts_tablet_manager.cc:1918] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:46.083626 2237 log.cc:1196] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/75d2028e0d9b44649bc95db345286ea1
I20260812 03:04:46.084507 2237 ts_tablet_manager.cc:1939] T 75d2028e0d9b44649bc95db345286ea1 P 3ee36e228a744157be9f55ea714b3109: Deleting consensus metadata
I20260812 03:04:46.087839 1359 catalog_manager.cc:4841] TS 3ee36e228a744157be9f55ea714b3109 (127.10.118.67:41137): tablet 75d2028e0d9b44649bc95db345286ea1 (table test-table [id=355f63fd8032476ab551fcbea1a43abb]) successfully deleted
I20260812 03:04:46.090926 2234 ts_tablet_manager.cc:1918] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:46.091425 2234 log.cc:1196] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/75d2028e0d9b44649bc95db345286ea1
I20260812 03:04:46.092500 2234 ts_tablet_manager.cc:1939] T 75d2028e0d9b44649bc95db345286ea1 P 69979f2d34b849758aab62eae84869d2: Deleting consensus metadata
W20260812 03:04:46.095505 1360 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:43309 (error 108)
I20260812 03:04:46.095709 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 1722
I20260812 03:04:46.100557 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:46.098903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43309 (local address 127.0.0.1:60724)
0812 03:04:46.099598 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:46.099904 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260812 03:04:46.101029 1360 catalog_manager.cc:4584] TS 16f66938a6e5438ba9a8929c66b21fef (127.10.118.65:43309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75d2028e0d9b44649bc95db345286ea1: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43309: connect: Connection refused (error 111)
I20260812 03:04:46.101270 1359 catalog_manager.cc:4841] TS 69979f2d34b849758aab62eae84869d2 (127.10.118.66:35983): tablet 75d2028e0d9b44649bc95db345286ea1 (table test-table [id=355f63fd8032476ab551fcbea1a43abb]) successfully deleted
I20260812 03:04:46.133023 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 1934
I20260812 03:04:46.157110 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:46.155516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43309 (local address 127.0.0.1:60736)
0812 03:04:46.156194 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:46.156455 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 03:04:46.159878 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:46.158428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43309 (local address 127.0.0.1:60750)
0812 03:04:46.159048 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:46.159264 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 03:04:46.172142 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 1322
2026-08-12T03:04:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (8234 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:04:46 dist-test-slave-zpvx krb5kdc[2241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:04:46 dist-test-slave-zpvx krb5kdc[2241](info): set up 2 sockets
Aug 12 03:04:46 dist-test-slave-zpvx krb5kdc[2241](info): commencing operation
krb5kdc: starting...
W20260812 03:04:47.653174 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.415s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:04:47 dist-test-slave-zpvx krb5kdc[2241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:04:47.889950 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:42147
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:42147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:48.198622 2261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:48.199278 2261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:48.199898 2261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:48.230949 2261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:04:48.231300 2261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:48.231600 2261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:48.231855 2261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:04:48.232098 2261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:04:48.268779 2261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:42147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:42147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:48.270061 2261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:48.271137 2261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:48.288271 2268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:48.288290 2270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:48.289958 2261 server_base.cc:1034] running on GCE node
W20260812 03:04:48.288787 2267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:49.472606 2261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:49.475229 2261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:49.476590 2261 hybrid_clock.cc:648] HybridClock initialized: now 1786503889476560 us; error 48 us; skew 500 ppm
I20260812 03:04:49.477453 2261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:49 dist-test-slave-zpvx krb5kdc[2241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503889, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:49.483927 2261 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:04:49.489599 2261 webserver.cc:460] Webserver started at http://127.10.118.126:41449/ using document root <none> and password file <none>
I20260812 03:04:49.490499 2261 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:49.490682 2261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:49.491093 2261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:49.496445 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "f4c2b7f116a942c19953e5ce90a247ea"
format_stamp: "Formatted at 2026-08-12 03:04:49 on dist-test-slave-zpvx"
server_key: "d43e8eaaab1f9bde06f26f7d9bb7bfa8"
server_key_iv: "1cc570becc8db836f7e952f1fd6d318c"
server_key_version: "encryptionkey@0"
I20260812 03:04:49.497572 2261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "f4c2b7f116a942c19953e5ce90a247ea"
format_stamp: "Formatted at 2026-08-12 03:04:49 on dist-test-slave-zpvx"
server_key: "d43e8eaaab1f9bde06f26f7d9bb7bfa8"
server_key_iv: "1cc570becc8db836f7e952f1fd6d318c"
server_key_version: "encryptionkey@0"
I20260812 03:04:49.504901 2261 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 03:04:49.510355 2293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:49.511394 2261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260812 03:04:49.511723 2261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "f4c2b7f116a942c19953e5ce90a247ea"
format_stamp: "Formatted at 2026-08-12 03:04:49 on dist-test-slave-zpvx"
server_key: "d43e8eaaab1f9bde06f26f7d9bb7bfa8"
server_key_iv: "1cc570becc8db836f7e952f1fd6d318c"
server_key_version: "encryptionkey@0"
I20260812 03:04:49.512071 2261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:49.564471 2261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:49.570093 2261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:49.570598 2261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:49.640220 2261 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:42147
I20260812 03:04:49.640308 2379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:42147 every 8 connection(s)
I20260812 03:04:49.642949 2261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:04:49.645519 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 2261
I20260812 03:04:49.646155 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:04:49.647274 10713 external_mini_cluster.cc:1428] Setting key fe14a4808135b1f42cd84557b19d9582
I20260812 03:04:49.649106 2384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:49.674786 2384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea: Bootstrap starting.
Aug 12 03:04:49 dist-test-slave-zpvx krb5kdc[2241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:49.680333 2384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:49.682687 2384 log.cc:824] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:49.688902 2384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea: No bootstrap required, opened a new log
I20260812 03:04:49.693713 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:49.649587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.10.118.126:42147)
0812 03:04:49.652197 (+ 2610us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:49.652242 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:49.652315 (+ 73us) server_negotiation.cc:407] Connection header received
0812 03:04:49.659460 (+ 7145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:49.659523 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:49.660043 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:49.660566 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:49.663103 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:49.665843 (+ 2740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:49.667742 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:49.669221 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:49.669556 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:49.679329 (+ 9773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:49.679364 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:49.679404 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:49.679555 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:49.685555 (+ 6000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:49.686983 (+ 1428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:49.687015 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:49.687044 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:49.687358 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:49.687968 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:49.687996 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:49.688019 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:49.689046 (+ 1027us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:49.689541 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:49.690144 (+ 603us) server_negotiation.cc:299] Negotiation successful
0812 03:04:49.690781 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2055,"thread_start_us":1607,"threads_started":1}
I20260812 03:04:49.711227 2384 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c2b7f116a942c19953e5ce90a247ea" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42147 } }
I20260812 03:04:49.711897 2384 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:49.712095 2384 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c2b7f116a942c19953e5ce90a247ea, State: Initialized, Role: FOLLOWER
I20260812 03:04:49.712860 2384 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c2b7f116a942c19953e5ce90a247ea" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42147 } }
I20260812 03:04:49.713366 2384 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:04:49.713586 2384 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:04:49.713841 2384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:49.717864 2384 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c2b7f116a942c19953e5ce90a247ea" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42147 } }
I20260812 03:04:49.718690 2384 leader_election.cc:302] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c2b7f116a942c19953e5ce90a247ea; no voters:
I20260812 03:04:49.720510 2384 leader_election.cc:288] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:04:49.720885 2396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:49.723450 2396 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [term 1 LEADER]: Becoming Leader. State: Replica: f4c2b7f116a942c19953e5ce90a247ea, State: Running, Role: LEADER
I20260812 03:04:49.724203 2396 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c2b7f116a942c19953e5ce90a247ea" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42147 } }
I20260812 03:04:49.725176 2384 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:04:49.735083 2399 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c2b7f116a942c19953e5ce90a247ea. Latest consensus state: current_term: 1 leader_uuid: "f4c2b7f116a942c19953e5ce90a247ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c2b7f116a942c19953e5ce90a247ea" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42147 } } }
I20260812 03:04:49.737066 2398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4c2b7f116a942c19953e5ce90a247ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c2b7f116a942c19953e5ce90a247ea" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 42147 } } }
I20260812 03:04:49.737735 2399 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:49.737879 2398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:49.740928 2412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:04:49.751684 2412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:04:49.767951 2412 catalog_manager.cc:1347] Generated new cluster ID: b650c92e9d81454897cb34838db70103
I20260812 03:04:49.768282 2412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:04:49.808238 2412 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:04:49.811230 2412 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:04:49.825157 2412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea: Generated new TSK 0
I20260812 03:04:49.826767 2412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:04:49.911089 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42147
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:50.234413 2426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:50.234941 2426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:50.235601 2426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:50.267058 2426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:50.267479 2426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:50.268464 2426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:04:50.303866 2426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:50.305109 2426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:50.306137 2426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:50.325968 2436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:50.328361 2426 server_base.cc:1034] running on GCE node
W20260812 03:04:50.327287 2435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:50.329310 2438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:51.504984 2426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:51.507756 2426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:51.509187 2426 hybrid_clock.cc:648] HybridClock initialized: now 1786503891509162 us; error 45 us; skew 500 ppm
I20260812 03:04:51.509989 2426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:51 dist-test-slave-zpvx krb5kdc[2241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503891, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:51.520973 2426 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:04:51.527031 2426 webserver.cc:460] Webserver started at http://127.10.118.65:40849/ using document root <none> and password file <none>
I20260812 03:04:51.527956 2426 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:51.528139 2426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:51.528609 2426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:51.534142 2426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "be84b53d099c4ff9b50eebfe047a6ec8"
format_stamp: "Formatted at 2026-08-12 03:04:51 on dist-test-slave-zpvx"
server_key: "5514cf8314967640ad7f65beb5e5f583"
server_key_iv: "0442f1fa9f41c2b66a214b72d8f7328e"
server_key_version: "encryptionkey@0"
I20260812 03:04:51.535252 2426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "be84b53d099c4ff9b50eebfe047a6ec8"
format_stamp: "Formatted at 2026-08-12 03:04:51 on dist-test-slave-zpvx"
server_key: "5514cf8314967640ad7f65beb5e5f583"
server_key_iv: "0442f1fa9f41c2b66a214b72d8f7328e"
server_key_version: "encryptionkey@0"
I20260812 03:04:51.543258 2426 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s
I20260812 03:04:51.548830 2451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:51.549829 2426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 03:04:51.550112 2426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "be84b53d099c4ff9b50eebfe047a6ec8"
format_stamp: "Formatted at 2026-08-12 03:04:51 on dist-test-slave-zpvx"
server_key: "5514cf8314967640ad7f65beb5e5f583"
server_key_iv: "0442f1fa9f41c2b66a214b72d8f7328e"
server_key_version: "encryptionkey@0"
I20260812 03:04:51.550438 2426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:51.599494 2426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:51.605654 2426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:51.606096 2426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:51.608692 2426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:51.612780 2426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:51.613041 2426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:51.613270 2426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:51.613397 2426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:51.747627 2426 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:34095
I20260812 03:04:51.747752 2660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:34095 every 8 connection(s)
I20260812 03:04:51.750180 2426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:04:51.756348 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 2426
I20260812 03:04:51.756925 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:04:51.758049 10713 external_mini_cluster.cc:1428] Setting key 7f3ee5a93ebc5c6a87554f949fcfdfa9
Aug 12 03:04:51 dist-test-slave-zpvx krb5kdc[2241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503891, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
I20260812 03:04:51.792631 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:51.754352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:33251 (local address 127.10.118.126:42147)
0812 03:04:51.757451 (+ 3099us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:51.757474 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:51.757775 (+ 301us) server_negotiation.cc:407] Connection header received
0812 03:04:51.763293 (+ 5518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:51.763357 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:51.763626 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:51.763816 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:51.767381 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:51.768945 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:51.773070 (+ 4125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:51.773992 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:51.774280 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:51.781204 (+ 6924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:51.781245 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:51.781278 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:51.781438 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:51.786151 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:51.787352 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:51.787390 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:51.787422 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:51.787714 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:51.788509 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:51.788540 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:51.788566 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:51.789358 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:51.789635 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:51.790738 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0812 03:04:51.791077 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2627,"thread_start_us":957,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:04:51.793366 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:51.754581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42147 (local address 127.10.118.65:33251)
0812 03:04:51.757559 (+ 2978us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:51.757627 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:51.762844 (+ 5217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:51.764176 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:51.764234 (+ 58us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:51.765547 (+ 1313us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:51.766951 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:51.767028 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:51.769335 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:51.769372 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:51.772817 (+ 3445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:51.772855 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:51.774807 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:51.774857 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:51.775465 (+ 608us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:51.776671 (+ 1206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:51.776782 (+ 111us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:51.780837 (+ 4055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:51.786352 (+ 5515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:51.786376 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:51.786403 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:51.787108 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:51.787912 (+ 804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:51.787932 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:51.787946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:51.788248 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:51.789607 (+ 1359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:51.789638 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:51.790083 (+ 445us) client_negotiation.cc:769] Sending connection context
0812 03:04:51.790473 (+ 390us) client_negotiation.cc:240] Negotiation successful
0812 03:04:51.790981 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2362,"thread_start_us":1526,"threads_started":1}
I20260812 03:04:51.795360 2662 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42147
I20260812 03:04:51.795780 2662 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:51.796777 2662 heartbeater.cc:511] Master 127.10.118.126:42147 requested a full tablet report, sending...
I20260812 03:04:51.799582 2323 ts_manager.cc:194] Registered new tserver with Master: be84b53d099c4ff9b50eebfe047a6ec8 (127.10.118.65:34095)
I20260812 03:04:51.801802 2323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:33251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:04:51.824301 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42147
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:52.130182 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:52.130645 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:52.131284 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:52.161845 2671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:52.162230 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:52.163149 2671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:04:52.198098 2671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:52.199309 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:52.200362 2671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:52.220166 2682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:52.805753 2662 heartbeater.cc:503] Master 127.10.118.126:42147 was elected leader, sending a full tablet report...
W20260812 03:04:52.220222 2685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:52.220635 2678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:52.222321 2671 server_base.cc:1034] running on GCE node
I20260812 03:04:53.401333 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:53.404424 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:53.406013 2671 hybrid_clock.cc:648] HybridClock initialized: now 1786503893405987 us; error 49 us; skew 500 ppm
I20260812 03:04:53.406888 2671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:53 dist-test-slave-zpvx krb5kdc[2241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:53.414286 2671 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:04:53.419999 2671 webserver.cc:460] Webserver started at http://127.10.118.66:42989/ using document root <none> and password file <none>
I20260812 03:04:53.421207 2671 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:53.421422 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:53.421871 2671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:53.427805 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "948e3f07781e4794bc375ef98a2616ed"
format_stamp: "Formatted at 2026-08-12 03:04:53 on dist-test-slave-zpvx"
server_key: "29333233903ac050b9d4e16906a6d628"
server_key_iv: "f8ebdb98012b3df154a5b49cad6ac579"
server_key_version: "encryptionkey@0"
I20260812 03:04:53.429095 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "948e3f07781e4794bc375ef98a2616ed"
format_stamp: "Formatted at 2026-08-12 03:04:53 on dist-test-slave-zpvx"
server_key: "29333233903ac050b9d4e16906a6d628"
server_key_iv: "f8ebdb98012b3df154a5b49cad6ac579"
server_key_version: "encryptionkey@0"
I20260812 03:04:53.437235 2671 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:04:53.443183 2699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:53.444339 2671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 03:04:53.444801 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "948e3f07781e4794bc375ef98a2616ed"
format_stamp: "Formatted at 2026-08-12 03:04:53 on dist-test-slave-zpvx"
server_key: "29333233903ac050b9d4e16906a6d628"
server_key_iv: "f8ebdb98012b3df154a5b49cad6ac579"
server_key_version: "encryptionkey@0"
I20260812 03:04:53.445221 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:53.482180 2671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:53.488322 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:53.488808 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:53.491339 2671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:53.495319 2671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:53.495507 2671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:53.495827 2671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:53.495987 2671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:53.635057 2671 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:34963
I20260812 03:04:53.635205 2897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:34963 every 8 connection(s)
I20260812 03:04:53.637679 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:04:53.641867 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 2671
I20260812 03:04:53.642645 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:04:53.643965 10713 external_mini_cluster.cc:1428] Setting key 03191819ba10ea7a93fecb432c8cfc02
Aug 12 03:04:53 dist-test-slave-zpvx krb5kdc[2241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503893, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:04:53.683494 2903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:53.644764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:44771 (local address 127.10.118.126:42147)
0812 03:04:53.647063 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:53.647094 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:53.649123 (+ 2029us) server_negotiation.cc:407] Connection header received
0812 03:04:53.656626 (+ 7503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:53.656655 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:53.656872 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:53.657047 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:53.660361 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:53.661790 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:53.665258 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:53.666000 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:53.666216 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:53.672509 (+ 6293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:53.672532 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:53.672548 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:53.672645 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:53.676570 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:53.677829 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:53.677881 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:53.677905 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:53.678160 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:53.678870 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:53.678903 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:53.678917 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:53.679407 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:53.679654 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:53.680742 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0812 03:04:53.681204 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1675,"thread_start_us":1372,"threads_started":1}
I20260812 03:04:53.684245 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:53.645259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42147 (local address 127.10.118.66:44771)
0812 03:04:53.648889 (+ 3630us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:53.648956 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:53.656200 (+ 7244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:53.657383 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:53.657426 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:53.658737 (+ 1311us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:53.660017 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:53.660089 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:53.662051 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:53.662086 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:53.664945 (+ 2859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:53.665018 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:53.666367 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:53.666413 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:53.667053 (+ 640us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:53.668120 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:53.668221 (+ 101us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:53.672121 (+ 3900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:53.676784 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:53.676828 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:53.676852 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:53.677610 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:53.678355 (+ 745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:53.678381 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:53.678407 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:53.678665 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:53.679616 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:53.679656 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:53.680094 (+ 438us) client_negotiation.cc:769] Sending connection context
0812 03:04:53.680503 (+ 409us) client_negotiation.cc:240] Negotiation successful
0812 03:04:53.681222 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3062,"thread_start_us":2239,"threads_started":1}
I20260812 03:04:53.686302 2898 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42147
I20260812 03:04:53.686713 2898 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:53.687651 2898 heartbeater.cc:511] Master 127.10.118.126:42147 requested a full tablet report, sending...
I20260812 03:04:53.689992 2323 ts_manager.cc:194] Registered new tserver with Master: 948e3f07781e4794bc375ef98a2616ed (127.10.118.66:34963)
I20260812 03:04:53.691430 2323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:44771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:04:53.709196 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42147
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:04:54.022840 2909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:54.023313 2909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:54.023905 2909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:54.054786 2909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:54.055121 2909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:54.056020 2909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:04:54.091251 2909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:42147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:54.092324 2909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:54.093310 2909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:54.112700 2918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:54.695166 2898 heartbeater.cc:503] Master 127.10.118.126:42147 was elected leader, sending a full tablet report...
W20260812 03:04:54.112999 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:54.114512 2909 server_base.cc:1034] running on GCE node
W20260812 03:04:54.116631 2919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:55.297835 2909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:55.300226 2909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:55.301707 2909 hybrid_clock.cc:648] HybridClock initialized: now 1786503895301663 us; error 62 us; skew 500 ppm
I20260812 03:04:55.302703 2909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:04:55 dist-test-slave-zpvx krb5kdc[2241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:04:55.309746 2909 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:04:55.315033 2909 webserver.cc:460] Webserver started at http://127.10.118.67:34669/ using document root <none> and password file <none>
I20260812 03:04:55.316021 2909 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:55.316277 2909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:55.316815 2909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:55.322129 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "87d422f965fb4805b52a09489ebff405"
format_stamp: "Formatted at 2026-08-12 03:04:55 on dist-test-slave-zpvx"
server_key: "2c188981462f74fbed90367d2cbd610a"
server_key_iv: "1201773c5f0a1ff5f10d18daab74e0cb"
server_key_version: "encryptionkey@0"
I20260812 03:04:55.323340 2909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "87d422f965fb4805b52a09489ebff405"
format_stamp: "Formatted at 2026-08-12 03:04:55 on dist-test-slave-zpvx"
server_key: "2c188981462f74fbed90367d2cbd610a"
server_key_iv: "1201773c5f0a1ff5f10d18daab74e0cb"
server_key_version: "encryptionkey@0"
I20260812 03:04:55.330804 2909 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260812 03:04:55.336778 2945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:55.337841 2909 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:04:55.338207 2909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "87d422f965fb4805b52a09489ebff405"
format_stamp: "Formatted at 2026-08-12 03:04:55 on dist-test-slave-zpvx"
server_key: "2c188981462f74fbed90367d2cbd610a"
server_key_iv: "1201773c5f0a1ff5f10d18daab74e0cb"
server_key_version: "encryptionkey@0"
I20260812 03:04:55.338589 2909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:55.374146 2909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:55.380111 2909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:55.380601 2909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:55.383296 2909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:04:55.387163 2909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:04:55.387404 2909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:55.387666 2909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:04:55.387846 2909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:55.523135 2909 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:44535
I20260812 03:04:55.523270 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:44535 every 8 connection(s)
I20260812 03:04:55.525848 2909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:04:55.536345 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 2909
I20260812 03:04:55.537003 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:04:55.538268 10713 external_mini_cluster.cc:1428] Setting key 0632a3ab6c055ed1c7ba1c5706974b20
Aug 12 03:04:55 dist-test-slave-zpvx krb5kdc[2241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503895, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:04:55.560815 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.530003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:36509 (local address 127.10.118.126:42147)
0812 03:04:55.531643 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:55.531670 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:55.533466 (+ 1796us) server_negotiation.cc:407] Connection header received
0812 03:04:55.538614 (+ 5148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:55.538644 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:55.538904 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:55.539108 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:55.541877 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.542993 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.545049 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.545681 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.545874 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.550508 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:55.550532 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:55.550549 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:55.550643 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:55.554473 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:55.555881 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:55.555913 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:55.555934 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:55.556194 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:55.557096 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:55.557151 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:55.557169 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:55.557786 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:55.557970 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:55.559008 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0812 03:04:55.559349 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":773,"threads_started":1}
I20260812 03:04:55.561385 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.530400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:42147 (local address 127.10.118.67:36509)
0812 03:04:55.533242 (+ 2842us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:55.533311 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:55.538223 (+ 4912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:55.539339 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:55.539374 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:55.540413 (+ 1039us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:04:55.541606 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.541651 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.543227 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.543250 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.544836 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.544873 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.545956 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.545980 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.546307 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.547186 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:04:55.547249 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:04:55.550211 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:04:55.554698 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:55.554730 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:55.554767 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:55.555638 (+ 871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:55.556440 (+ 802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:04:55.556468 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:04:55.556490 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:04:55.556852 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:04:55.557967 (+ 1115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:04:55.557998 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:04:55.558419 (+ 421us) client_negotiation.cc:769] Sending connection context
0812 03:04:55.558812 (+ 393us) client_negotiation.cc:240] Negotiation successful
0812 03:04:55.559286 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2325,"thread_start_us":1570,"threads_started":1}
I20260812 03:04:55.563000 3151 heartbeater.cc:347] Connected to a master server at 127.10.118.126:42147
I20260812 03:04:55.563369 3151 heartbeater.cc:464] Registering TS with master...
I20260812 03:04:55.564210 3151 heartbeater.cc:511] Master 127.10.118.126:42147 requested a full tablet report, sending...
I20260812 03:04:55.566284 2323 ts_manager.cc:194] Registered new tserver with Master: 87d422f965fb4805b52a09489ebff405 (127.10.118.67:44535)
I20260812 03:04:55.567752 2323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:36509
I20260812 03:04:55.571718 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:04:55.609711 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.586968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.10.118.126:42147)
0812 03:04:55.587404 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:55.587422 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:55.588582 (+ 1160us) server_negotiation.cc:407] Connection header received
0812 03:04:55.589018 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:55.589037 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:55.589210 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:55.589382 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:55.591809 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.592768 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.595271 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.595846 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.596128 (+ 282us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.600149 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:55.600172 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:55.600191 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:04:55.600289 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:55.603604 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:55.605059 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:55.605079 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:55.605093 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:55.605291 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:04:55.606126 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:04:55.606149 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:04:55.606166 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:04:55.606654 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:55.607095 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:55.607922 (+ 827us) server_negotiation.cc:299] Negotiation successful
0812 03:04:55.608301 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260812 03:04:55.615916 2323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:04:55.636979 2323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:04:55.704470 3189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.670696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:44535 (local address 127.0.0.1:56746)
0812 03:04:55.683090 (+ 12394us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:55.683183 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:55.683529 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:55.684479 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:55.684510 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:55.684625 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:55.685491 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.685544 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.693110 (+ 7566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.693140 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.696025 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.696073 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.701053 (+ 4980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.701086 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.701556 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.701908 (+ 352us) client_negotiation.cc:769] Sending connection context
0812 03:04:55.702145 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 03:04:55.702562 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11924,"mutex_wait_us":106,"thread_start_us":20222,"threads_started":1}
I20260812 03:04:55.704923 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.672648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.10.118.67:44535)
0812 03:04:55.681939 (+ 9291us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:55.681982 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:55.683759 (+ 1777us) server_negotiation.cc:407] Connection header received
0812 03:04:55.683912 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:55.683933 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:55.684183 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:55.684405 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:55.689444 (+ 5039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.692799 (+ 3355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.696526 (+ 3727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.698522 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.698815 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.699266 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:55.702851 (+ 3585us) server_negotiation.cc:299] Negotiation successful
0812 03:04:55.703162 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8634,"thread_start_us":8158,"threads_started":1}
I20260812 03:04:55.710898 3031 tablet_service.cc:1467] Processing CreateTablet for tablet 7217dc1265a34955b23f06cd40f7bb66 (DEFAULT_TABLE table=test-table [id=8cbbe0ba49b34deea9cfefff1d166196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:55.713260 3031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7217dc1265a34955b23f06cd40f7bb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:55.715480 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.673600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.10.118.65:34095)
0812 03:04:55.700289 (+ 26689us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:55.700321 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:55.700410 (+ 89us) server_negotiation.cc:407] Connection header received
0812 03:04:55.700591 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:55.700618 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:55.701006 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:55.701245 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:55.703112 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.705551 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.709947 (+ 4396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.711809 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.712142 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.712652 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:55.713533 (+ 881us) server_negotiation.cc:299] Negotiation successful
0812 03:04:55.713825 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26189,"thread_start_us":26963,"threads_started":1}
I20260812 03:04:55.715526 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.673317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:34095 (local address 127.0.0.1:43036)
0812 03:04:55.690245 (+ 16928us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:55.690599 (+ 354us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:55.690864 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:55.701350 (+ 10486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:55.701395 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:55.701540 (+ 145us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:55.702678 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.702736 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.706689 (+ 3953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.706729 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.709592 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.709646 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.712146 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.712179 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.713045 (+ 866us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.713147 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 03:04:55.713311 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 03:04:55.713681 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16272,"mutex_wait_us":416,"thread_start_us":9029,"threads_started":1}
I20260812 03:04:55.721206 2549 tablet_service.cc:1467] Processing CreateTablet for tablet 7217dc1265a34955b23f06cd40f7bb66 (DEFAULT_TABLE table=test-table [id=8cbbe0ba49b34deea9cfefff1d166196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:55.723291 2549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7217dc1265a34955b23f06cd40f7bb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:55.727473 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.692472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.10.118.66:34963)
0812 03:04:55.702654 (+ 10182us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:55.702688 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:55.705752 (+ 3064us) server_negotiation.cc:407] Connection header received
0812 03:04:55.706202 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:55.706238 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:55.706611 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:55.706820 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:55.708272 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.710196 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.714149 (+ 3953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.717195 (+ 3046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.717527 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.718053 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:55.725156 (+ 7103us) server_negotiation.cc:299] Negotiation successful
0812 03:04:55.725509 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9697,"thread_start_us":9261,"threads_started":1}
I20260812 03:04:55.728536 3189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.698393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:34963 (local address 127.0.0.1:51716)
0812 03:04:55.705572 (+ 7179us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:55.705625 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:55.705852 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:55.707362 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:55.707388 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:55.707494 (+ 106us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:55.707908 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.707952 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.710463 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.710496 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.713740 (+ 3244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.713808 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.722036 (+ 8228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.722080 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.722542 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.722723 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 03:04:55.722953 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 03:04:55.723294 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6786,"mutex_wait_us":284,"thread_start_us":7601,"threads_started":1}
I20260812 03:04:55.733330 2785 tablet_service.cc:1467] Processing CreateTablet for tablet 7217dc1265a34955b23f06cd40f7bb66 (DEFAULT_TABLE table=test-table [id=8cbbe0ba49b34deea9cfefff1d166196]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:04:55.735189 2785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7217dc1265a34955b23f06cd40f7bb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:55.745350 3195 tablet_bootstrap.cc:492] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Bootstrap starting.
I20260812 03:04:55.752183 3195 tablet_bootstrap.cc:654] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:55.754745 3195 log.cc:824] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:55.767737 3196 tablet_bootstrap.cc:492] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8: Bootstrap starting.
I20260812 03:04:55.771487 3198 tablet_bootstrap.cc:492] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Bootstrap starting.
I20260812 03:04:55.776223 3195 tablet_bootstrap.cc:492] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: No bootstrap required, opened a new log
I20260812 03:04:55.776794 3195 ts_tablet_manager.cc:1397] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.012s
I20260812 03:04:55.778864 3196 tablet_bootstrap.cc:654] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:55.779681 3198 tablet_bootstrap.cc:654] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:55.781807 3196 log.cc:824] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:55.782572 3198 log.cc:824] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:55.788457 3196 tablet_bootstrap.cc:492] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8: No bootstrap required, opened a new log
I20260812 03:04:55.788934 3196 ts_tablet_manager.cc:1397] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20260812 03:04:55.789427 3198 tablet_bootstrap.cc:492] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: No bootstrap required, opened a new log
I20260812 03:04:55.789938 3198 ts_tablet_manager.cc:1397] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20260812 03:04:55.807190 3195 raft_consensus.cc:348] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.808142 3195 raft_consensus.cc:374] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:55.808506 3195 raft_consensus.cc:729] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d422f965fb4805b52a09489ebff405, State: Initialized, Role: FOLLOWER
I20260812 03:04:55.809593 3195 consensus_queue.cc:260] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.811928 3196 raft_consensus.cc:348] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.812934 3196 raft_consensus.cc:374] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:55.812988 3151 heartbeater.cc:503] Master 127.10.118.126:42147 was elected leader, sending a full tablet report...
I20260812 03:04:55.813320 3196 raft_consensus.cc:729] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be84b53d099c4ff9b50eebfe047a6ec8, State: Initialized, Role: FOLLOWER
I20260812 03:04:55.813920 3195 ts_tablet_manager.cc:1428] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Time spent starting tablet: real 0.037s user 0.034s sys 0.002s
I20260812 03:04:55.814352 3196 consensus_queue.cc:260] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.820199 3196 ts_tablet_manager.cc:1428] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8: Time spent starting tablet: real 0.031s user 0.021s sys 0.008s
I20260812 03:04:55.819748 3198 raft_consensus.cc:348] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.820680 3198 raft_consensus.cc:374] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:55.820986 3198 raft_consensus.cc:729] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948e3f07781e4794bc375ef98a2616ed, State: Initialized, Role: FOLLOWER
I20260812 03:04:55.821913 3198 consensus_queue.cc:260] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.826151 3198 ts_tablet_manager.cc:1428] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Time spent starting tablet: real 0.036s user 0.032s sys 0.004s
I20260812 03:04:55.892325 3204 raft_consensus.cc:482] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:04:55.892884 3204 raft_consensus.cc:504] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
W20260812 03:04:55.894892 2900 tablet.cc:2369] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:55.896522 3204 leader_election.cc:288] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 948e3f07781e4794bc375ef98a2616ed (127.10.118.66:34963), be84b53d099c4ff9b50eebfe047a6ec8 (127.10.118.65:34095)
I20260812 03:04:55.906203 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.896165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:56853 (local address 127.10.118.66:34963)
0812 03:04:55.896824 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:55.896852 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:55.896913 (+ 61us) server_negotiation.cc:407] Connection header received
0812 03:04:55.897349 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:55.897368 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:55.897550 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:55.897723 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:55.898651 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.899914 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.902526 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.903693 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.903947 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.904117 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:55.904918 (+ 801us) server_negotiation.cc:299] Negotiation successful
0812 03:04:55.905117 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260812 03:04:55.906399 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.896064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:34963 (local address 127.10.118.67:56853)
0812 03:04:55.896775 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:55.896821 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:55.897075 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:55.897767 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:55.897800 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:55.897910 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:55.898317 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.898361 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.900162 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.900196 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.902257 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.902293 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.904009 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.904041 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.904564 (+ 523us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.904707 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 03:04:55.904877 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 03:04:55.905173 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":99}
I20260812 03:04:55.908246 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7217dc1265a34955b23f06cd40f7bb66" candidate_uuid: "87d422f965fb4805b52a09489ebff405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948e3f07781e4794bc375ef98a2616ed" is_pre_election: true
I20260812 03:04:55.909219 2821 raft_consensus.cc:2454] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d422f965fb4805b52a09489ebff405 in term 0.
I20260812 03:04:55.909777 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.896644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:36183 (local address 127.10.118.65:34095)
0812 03:04:55.897203 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:55.897231 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:55.899461 (+ 2230us) server_negotiation.cc:407] Connection header received
0812 03:04:55.899879 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:55.899908 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:55.900187 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:55.900341 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:04:55.901811 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.902775 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.905731 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.907311 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.907643 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.907810 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:55.908568 (+ 758us) server_negotiation.cc:299] Negotiation successful
0812 03:04:55.908760 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260812 03:04:55.910614 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:55.897098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:34095 (local address 127.10.118.67:36183)
0812 03:04:55.899286 (+ 2188us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:55.899322 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 03:04:55.899598 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:04:55.900424 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:04:55.900451 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:04:55.900548 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:04:55.901538 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.901582 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.903024 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.903048 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.905409 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:04:55.905462 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:04:55.907704 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:04:55.907735 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:04:55.908146 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:55.908311 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 03:04:55.908491 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 03:04:55.908787 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1787,"mutex_wait_us":102,"thread_start_us":1471,"threads_started":1}
I20260812 03:04:55.910605 2950 leader_election.cc:302] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d422f965fb4805b52a09489ebff405, 948e3f07781e4794bc375ef98a2616ed; no voters:
I20260812 03:04:55.911525 3204 raft_consensus.cc:2781] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:04:55.911842 3204 raft_consensus.cc:482] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:04:55.912217 3204 raft_consensus.cc:3037] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:55.912456 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7217dc1265a34955b23f06cd40f7bb66" candidate_uuid: "87d422f965fb4805b52a09489ebff405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" is_pre_election: true
I20260812 03:04:55.913378 2587 raft_consensus.cc:2454] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d422f965fb4805b52a09489ebff405 in term 0.
I20260812 03:04:55.917692 3204 raft_consensus.cc:504] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.918980 3204 leader_election.cc:288] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [CANDIDATE]: Term 1 election: Requested vote from peers 948e3f07781e4794bc375ef98a2616ed (127.10.118.66:34963), be84b53d099c4ff9b50eebfe047a6ec8 (127.10.118.65:34095)
I20260812 03:04:55.919808 2821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7217dc1265a34955b23f06cd40f7bb66" candidate_uuid: "87d422f965fb4805b52a09489ebff405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "948e3f07781e4794bc375ef98a2616ed"
I20260812 03:04:55.919962 2587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7217dc1265a34955b23f06cd40f7bb66" candidate_uuid: "87d422f965fb4805b52a09489ebff405" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be84b53d099c4ff9b50eebfe047a6ec8"
I20260812 03:04:55.920428 2821 raft_consensus.cc:3037] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:55.920464 2587 raft_consensus.cc:3037] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:55.927366 2821 raft_consensus.cc:2454] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d422f965fb4805b52a09489ebff405 in term 1.
I20260812 03:04:55.927366 2587 raft_consensus.cc:2454] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d422f965fb4805b52a09489ebff405 in term 1.
I20260812 03:04:55.928436 2951 leader_election.cc:302] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d422f965fb4805b52a09489ebff405, be84b53d099c4ff9b50eebfe047a6ec8; no voters:
I20260812 03:04:55.929088 3204 raft_consensus.cc:2781] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:55.930616 3204 raft_consensus.cc:686] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 1 LEADER]: Becoming Leader. State: Replica: 87d422f965fb4805b52a09489ebff405, State: Running, Role: LEADER
I20260812 03:04:55.931525 3204 consensus_queue.cc:237] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } }
I20260812 03:04:55.943354 2319 catalog_manager.cc:5495] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d422f965fb4805b52a09489ebff405 (127.10.118.67). New cstate: current_term: 1 leader_uuid: "87d422f965fb4805b52a09489ebff405" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d422f965fb4805b52a09489ebff405" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 } health_report { overall_health: UNKNOWN } } }
W20260812 03:04:56.009691 2663 tablet.cc:2369] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:56.027653 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:56.018669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.10.118.67:44535)
0812 03:04:56.019105 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:56.019124 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:56.019160 (+ 36us) server_negotiation.cc:407] Connection header received
0812 03:04:56.019328 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:56.019348 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:56.019593 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:56.019737 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:04:56.020563 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:56.021444 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:56.023356 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:56.023927 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:56.024136 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:56.025040 (+ 904us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:04:56.025535 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:04:56.025735 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:56.026360 (+ 625us) server_negotiation.cc:299] Negotiation successful
0812 03:04:56.026472 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260812 03:04:56.030728 3152 tablet.cc:2369] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:04:56.040277 2587 raft_consensus.cc:1261] T 7217dc1265a34955b23f06cd40f7bb66 P be84b53d099c4ff9b50eebfe047a6ec8 [term 1 FOLLOWER]: Refusing update from remote peer 87d422f965fb4805b52a09489ebff405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:04:56.040742 2821 raft_consensus.cc:1261] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 1 FOLLOWER]: Refusing update from remote peer 87d422f965fb4805b52a09489ebff405: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:04:56.042054 3213 consensus_queue.cc:1035] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be84b53d099c4ff9b50eebfe047a6ec8" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:04:56.042697 3204 consensus_queue.cc:1035] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [LEADER]: Connected to new peer: Peer: permanent_uuid: "948e3f07781e4794bc375ef98a2616ed" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 34963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:04:56.095191 2319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:56.095832 2319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:04:56.105612 2319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4c2b7f116a942c19953e5ce90a247ea: Sending DeleteTablet for 3 replicas of tablet 7217dc1265a34955b23f06cd40f7bb66
I20260812 03:04:56.107720 3031 tablet_service.cc:1514] Processing DeleteTablet for tablet 7217dc1265a34955b23f06cd40f7bb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:56 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:56746
I20260812 03:04:56.108412 2785 tablet_service.cc:1514] Processing DeleteTablet for tablet 7217dc1265a34955b23f06cd40f7bb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:56 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260812 03:04:56.111375 2549 tablet_service.cc:1514] Processing DeleteTablet for tablet 7217dc1265a34955b23f06cd40f7bb66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:04:56 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:43036
I20260812 03:04:56.112002 3233 tablet_replica.cc:330] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: stopping tablet replica
I20260812 03:04:56.112941 3233 raft_consensus.cc:2229] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:04:56.114334 3233 raft_consensus.cc:2258] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:56.120509 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 2426
I20260812 03:04:56.125964 3234 tablet_replica.cc:330] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: stopping tablet replica
I20260812 03:04:56.127031 3234 raft_consensus.cc:2229] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:04:56.127751 3234 raft_consensus.cc:2258] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:04:56.135221 3233 ts_tablet_manager.cc:1905] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:56.135251 3234 ts_tablet_manager.cc:1905] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:04:56.150415 3234 ts_tablet_manager.cc:1918] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:56.150929 3234 log.cc:1196] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/7217dc1265a34955b23f06cd40f7bb66
I20260812 03:04:56.151824 3234 ts_tablet_manager.cc:1939] T 7217dc1265a34955b23f06cd40f7bb66 P 948e3f07781e4794bc375ef98a2616ed: Deleting consensus metadata
I20260812 03:04:56.152908 3233 ts_tablet_manager.cc:1918] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:04:56.153415 3233 log.cc:1196] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/7217dc1265a34955b23f06cd40f7bb66
I20260812 03:04:56.154317 3233 ts_tablet_manager.cc:1939] T 7217dc1265a34955b23f06cd40f7bb66 P 87d422f965fb4805b52a09489ebff405: Deleting consensus metadata
I20260812 03:04:56.155304 2294 catalog_manager.cc:4841] TS 948e3f07781e4794bc375ef98a2616ed (127.10.118.66:34963): tablet 7217dc1265a34955b23f06cd40f7bb66 (table test-table [id=8cbbe0ba49b34deea9cfefff1d166196]) successfully deleted
I20260812 03:04:56.157405 2294 catalog_manager.cc:4841] TS 87d422f965fb4805b52a09489ebff405 (127.10.118.67:44535): tablet 7217dc1265a34955b23f06cd40f7bb66 (table test-table [id=8cbbe0ba49b34deea9cfefff1d166196]) successfully deleted
W20260812 03:04:56.166289 2299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:34095 (error 108)
I20260812 03:04:56.167447 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 2671
I20260812 03:04:56.174393 3189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:56.172968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:34095 (local address 127.0.0.1:43046)
0812 03:04:56.173657 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0812 03:04:56.173871 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:34095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309}
W20260812 03:04:56.174917 2299 catalog_manager.cc:4584] TS be84b53d099c4ff9b50eebfe047a6ec8 (127.10.118.65:34095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7217dc1265a34955b23f06cd40f7bb66: Network error: Client connection negotiation failed: client connection to 127.10.118.65:34095: connect: Connection refused (error 111)
I20260812 03:04:56.197296 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 2909
I20260812 03:04:56.226998 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 2261
2026-08-12T03:04:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (10058 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T03:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:04:56Z Disabled control of system clock
I20260812 03:04:56.328650 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:46513
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:46513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 03:04:56.633101 3247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:56.633754 3247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:56.634322 3247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:56.664723 3247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:04:56.665086 3247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:56.665321 3247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:56.665583 3247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:04:56.665834 3247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:04:56.705611 3247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:46513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:46513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:56.706851 3247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:56.708535 3247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:56.725394 3261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:56.725412 3260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:56.725996 3266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:56.727269 3247 server_base.cc:1034] running on GCE node
I20260812 03:04:57.914891 3247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:57.918177 3247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:57.919638 3247 hybrid_clock.cc:648] HybridClock initialized: now 1786503897919603 us; error 44 us; skew 500 ppm
I20260812 03:04:57.920495 3247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:04:57.932808 3247 webserver.cc:460] Webserver started at http://127.10.118.126:33561/ using document root <none> and password file <none>
I20260812 03:04:57.933719 3247 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:57.933904 3247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:57.934335 3247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:57.939545 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "8dd42023b5f24f118238bbff7e5dc691"
format_stamp: "Formatted at 2026-08-12 03:04:57 on dist-test-slave-zpvx"
server_key: "cba8a2cf7bf9a54e5ae072e136dbe0df"
server_key_iv: "20a33f4694b4492b1f0e0a0735514944"
server_key_version: "encryptionkey@0"
I20260812 03:04:57.940681 3247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "8dd42023b5f24f118238bbff7e5dc691"
format_stamp: "Formatted at 2026-08-12 03:04:57 on dist-test-slave-zpvx"
server_key: "cba8a2cf7bf9a54e5ae072e136dbe0df"
server_key_iv: "20a33f4694b4492b1f0e0a0735514944"
server_key_version: "encryptionkey@0"
I20260812 03:04:57.947765 3247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 03:04:57.953362 3277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:57.954406 3247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 03:04:57.954734 3247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "8dd42023b5f24f118238bbff7e5dc691"
format_stamp: "Formatted at 2026-08-12 03:04:57 on dist-test-slave-zpvx"
server_key: "cba8a2cf7bf9a54e5ae072e136dbe0df"
server_key_iv: "20a33f4694b4492b1f0e0a0735514944"
server_key_version: "encryptionkey@0"
I20260812 03:04:57.955106 3247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:04:57.992024 3247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:04:57.993526 3247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:04:57.993990 3247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:04:58.062799 3247 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:46513
I20260812 03:04:58.062906 3373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:46513 every 8 connection(s)
I20260812 03:04:58.065596 3247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:04:58.070276 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 3247
I20260812 03:04:58.070712 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:04:58.070868 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:04:58.071803 10713 external_mini_cluster.cc:1428] Setting key e18288e551d38f6470ca58cb1cf1caf5
I20260812 03:04:58.097412 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:04:58.074114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57778 (local address 127.10.118.126:46513)
0812 03:04:58.077070 (+ 2956us) server_negotiation.cc:206] Beginning negotiation
0812 03:04:58.077106 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 03:04:58.077181 (+ 75us) server_negotiation.cc:407] Connection header received
0812 03:04:58.082174 (+ 4993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:04:58.082231 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:04:58.082757 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:04:58.083214 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:04:58.084328 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:58.086929 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:58.089237 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:04:58.090583 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:04:58.090948 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:04:58.092575 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:04:58.092611 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:04:58.092656 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:04:58.093704 (+ 1048us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:04:58.093835 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:04:58.093899 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:04:58.094092 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 03:04:58.094488 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 03:04:58.094865 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2283,"thread_start_us":1716,"threads_started":1}
I20260812 03:04:58.099982 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691: Bootstrap starting.
I20260812 03:04:58.105407 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691: Neither blocks nor log segments found. Creating new log.
I20260812 03:04:58.107101 3375 log.cc:824] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691: Log is configured to *not* fsync() on all Append() calls
I20260812 03:04:58.112229 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691: No bootstrap required, opened a new log
I20260812 03:04:58.133152 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd42023b5f24f118238bbff7e5dc691" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 46513 } }
I20260812 03:04:58.133864 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:04:58.134148 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd42023b5f24f118238bbff7e5dc691, State: Initialized, Role: FOLLOWER
I20260812 03:04:58.134861 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd42023b5f24f118238bbff7e5dc691" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 46513 } }
I20260812 03:04:58.135423 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:04:58.135701 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:04:58.136022 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:04:58.140076 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd42023b5f24f118238bbff7e5dc691" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 46513 } }
I20260812 03:04:58.140832 3375 leader_election.cc:302] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dd42023b5f24f118238bbff7e5dc691; no voters:
I20260812 03:04:58.142522 3375 leader_election.cc:288] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:04:58.143484 3384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:04:58.145995 3384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [term 1 LEADER]: Becoming Leader. State: Replica: 8dd42023b5f24f118238bbff7e5dc691, State: Running, Role: LEADER
I20260812 03:04:58.146703 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:04:58.146736 3384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd42023b5f24f118238bbff7e5dc691" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 46513 } }
I20260812 03:04:58.158289 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dd42023b5f24f118238bbff7e5dc691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd42023b5f24f118238bbff7e5dc691" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 46513 } } }
I20260812 03:04:58.158509 3387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dd42023b5f24f118238bbff7e5dc691. Latest consensus state: current_term: 1 leader_uuid: "8dd42023b5f24f118238bbff7e5dc691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dd42023b5f24f118238bbff7e5dc691" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 46513 } } }
I20260812 03:04:58.159301 3387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:58.159274 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691 [sys.catalog]: This master's current role is: LEADER
I20260812 03:04:58.163151 3404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:04:58.174141 3404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:04:58.192751 3404 catalog_manager.cc:1347] Generated new cluster ID: dcbda3a48e4348b9b3d22271bd04e407
I20260812 03:04:58.193096 3404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:04:58.286811 3404 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:04:58.288827 3404 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:04:58.302239 3404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691: Generated new TSK 0
I20260812 03:04:58.303408 3404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:04:58.327257 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:46513
--builtin_ntp_servers=127.10.118.84:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 03:04:58.645769 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:04:58.646301 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:04:58.646941 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:04:58.679071 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:04:58.679389 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:04:58.680253 3421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:04:58.724165 3421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:46513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:04:58.725502 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:04:58.727372 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:04:58.747943 3438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:58.749078 3433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:04:58.749953 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:04:58.751605 3421 server_base.cc:1034] running on GCE node
I20260812 03:04:59.942979 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:04:59.946297 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:04:59.948211 3421 hybrid_clock.cc:648] HybridClock initialized: now 1786503899948173 us; error 52 us; skew 500 ppm
I20260812 03:04:59.949334 3421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:04:59.956665 3421 webserver.cc:460] Webserver started at http://127.10.118.65:32911/ using document root <none> and password file <none>
I20260812 03:04:59.957830 3421 fs_manager.cc:359] Metadata directory not provided
I20260812 03:04:59.958103 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:04:59.958702 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:04:59.965279 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "fa99fe9561684fcc9933d36590057442"
format_stamp: "Formatted at 2026-08-12 03:04:59 on dist-test-slave-zpvx"
server_key: "2239d1e843ef9e3edb5fd69842b7bcf7"
server_key_iv: "577e447a402086e9999dec76610d47e3"
server_key_version: "encryptionkey@0"
I20260812 03:04:59.966981 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "fa99fe9561684fcc9933d36590057442"
format_stamp: "Formatted at 2026-08-12 03:04:59 on dist-test-slave-zpvx"
server_key: "2239d1e843ef9e3edb5fd69842b7bcf7"
server_key_iv: "577e447a402086e9999dec76610d47e3"
server_key_version: "encryptionkey@0"
I20260812 03:04:59.977119 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260812 03:04:59.985255 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:04:59.986581 3421 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260812 03:04:59.987033 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "fa99fe9561684fcc9933d36590057442"
format_stamp: "Formatted at 2026-08-12 03:04:59 on dist-test-slave-zpvx"
server_key: "2239d1e843ef9e3edb5fd69842b7bcf7"
server_key_iv: "577e447a402086e9999dec76610d47e3"
server_key_version: "encryptionkey@0"
I20260812 03:04:59.987596 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:00.027392 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:00.028837 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:00.029265 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:00.031718 3421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:00.035849 3421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:00.036032 3421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:00.036360 3421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:00.036545 3421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 03:05:00.176414 3421 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:38523
I20260812 03:05:00.176513 3654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:38523 every 8 connection(s)
I20260812 03:05:00.178985 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:05:00.185191 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 3421
I20260812 03:05:00.185845 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:05:00.187168 10713 external_mini_cluster.cc:1428] Setting key 0813fbc269c5b414f175fcb2689d96dd
I20260812 03:05:00.192795 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:46513
--builtin_ntp_servers=127.10.118.84:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 03:05:00.206103 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:00.183117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:37845 (local address 127.10.118.126:46513)
0812 03:05:00.184595 (+ 1478us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:00.184615 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:00.187277 (+ 2662us) server_negotiation.cc:407] Connection header received
0812 03:05:00.193114 (+ 5837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:00.193146 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:00.193343 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:00.193517 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:00.195423 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:00.196930 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:00.199935 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:00.201067 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:00.201366 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:00.203311 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:00.203334 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:00.203352 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:00.203396 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:00.203458 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:00.203494 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:00.203635 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:00.203969 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 03:05:00.204186 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":627,"threads_started":1}
I20260812 03:05:00.206930 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:00.183526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:46513 (local address 127.10.118.65:37845)
0812 03:05:00.187041 (+ 3515us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:00.187118 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:00.192791 (+ 5673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:00.193783 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:00.193817 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:00.193974 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:00.195096 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:00.195156 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:00.197227 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:00.197260 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:00.199636 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:00.199691 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:00.201419 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:00.201455 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:00.201889 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:00.202803 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:05:00.202820 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:00.202881 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:05:00.202991 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:00.203667 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:00.203717 (+ 50us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:00.203739 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 03:05:00.203922 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 03:05:00.204401 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2939,"thread_start_us":2080,"threads_started":1}
I20260812 03:05:00.208916 3656 heartbeater.cc:347] Connected to a master server at 127.10.118.126:46513
I20260812 03:05:00.209462 3656 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:00.210848 3656 heartbeater.cc:511] Master 127.10.118.126:46513 requested a full tablet report, sending...
I20260812 03:05:00.213768 3306 ts_manager.cc:194] Registered new tserver with Master: fa99fe9561684fcc9933d36590057442 (127.10.118.65:38523)
I20260812 03:05:00.215729 3306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.65:37845
W20260812 03:05:00.501578 3662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:00.502144 3662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:00.502728 3662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:00.534401 3662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:00.534642 3662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:00.535619 3662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:05:00.570644 3662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:46513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:00.571863 3662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:00.573520 3662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:00.591694 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:01.220000 3656 heartbeater.cc:503] Master 127.10.118.126:46513 was elected leader, sending a full tablet report...
W20260812 03:05:00.591765 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:00.591838 3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:00.593598 3662 server_base.cc:1034] running on GCE node
I20260812 03:05:01.770545 3662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:01.772995 3662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:01.774372 3662 hybrid_clock.cc:648] HybridClock initialized: now 1786503901774341 us; error 51 us; skew 500 ppm
I20260812 03:05:01.775318 3662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:05:01.781976 3662 webserver.cc:460] Webserver started at http://127.10.118.66:45003/ using document root <none> and password file <none>
I20260812 03:05:01.782981 3662 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:01.783202 3662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:01.783708 3662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:01.789212 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "4bc96d09a3a249f598ca4ba0c31df623"
format_stamp: "Formatted at 2026-08-12 03:05:01 on dist-test-slave-zpvx"
server_key: "d8d5ec881e080dcdd90340d700751de4"
server_key_iv: "3844d4e793ff7fb6754ab820db7e6c43"
server_key_version: "encryptionkey@0"
I20260812 03:05:01.790473 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "4bc96d09a3a249f598ca4ba0c31df623"
format_stamp: "Formatted at 2026-08-12 03:05:01 on dist-test-slave-zpvx"
server_key: "d8d5ec881e080dcdd90340d700751de4"
server_key_iv: "3844d4e793ff7fb6754ab820db7e6c43"
server_key_version: "encryptionkey@0"
I20260812 03:05:01.799865 3662 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260812 03:05:01.808171 3693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:01.809649 3662 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260812 03:05:01.810037 3662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "4bc96d09a3a249f598ca4ba0c31df623"
format_stamp: "Formatted at 2026-08-12 03:05:01 on dist-test-slave-zpvx"
server_key: "d8d5ec881e080dcdd90340d700751de4"
server_key_iv: "3844d4e793ff7fb6754ab820db7e6c43"
server_key_version: "encryptionkey@0"
I20260812 03:05:01.810437 3662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:01.873344 3662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:01.875059 3662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:01.875566 3662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:01.878271 3662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:01.882797 3662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:01.882983 3662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:01.883180 3662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:01.883304 3662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:02.020263 3662 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:46777
I20260812 03:05:02.020437 3898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:46777 every 8 connection(s)
I20260812 03:05:02.023629 3662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:05:02.024605 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 3662
I20260812 03:05:02.025034 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:05:02.026113 10713 external_mini_cluster.cc:1428] Setting key f2ffc6a2342227e7f3296afd2a5f37ce
I20260812 03:05:02.033280 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:46513
--builtin_ntp_servers=127.10.118.84:45331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 03:05:02.049822 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:02.029327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:55345 (local address 127.10.118.126:46513)
0812 03:05:02.030850 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:02.030891 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:02.032783 (+ 1892us) server_negotiation.cc:407] Connection header received
0812 03:05:02.037780 (+ 4997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:02.037815 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:02.038020 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:02.038209 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:02.039911 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:02.040937 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:02.043899 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:02.044577 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:02.044875 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:02.046854 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:02.046874 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:02.046890 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:02.046995 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:02.047127 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:02.047181 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:02.047337 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:02.047661 (+ 324us) server_negotiation.cc:299] Negotiation successful
0812 03:05:02.047894 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":550,"threads_started":1}
I20260812 03:05:02.050756 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:02.029721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:46513 (local address 127.10.118.66:55345)
0812 03:05:02.032559 (+ 2838us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:02.032632 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:02.037395 (+ 4763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:02.038516 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:02.038554 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:02.038698 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:02.039619 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:02.039662 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:02.041192 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:02.041213 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:02.043625 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:02.043664 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:02.044933 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:02.044965 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:02.045402 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:02.046324 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:05:02.046342 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:02.046405 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:05:02.046523 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:02.047345 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:02.047398 (+ 53us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:02.047422 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 03:05:02.047620 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 03:05:02.048106 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2176,"thread_start_us":1447,"threads_started":1}
I20260812 03:05:02.052778 3903 heartbeater.cc:347] Connected to a master server at 127.10.118.126:46513
I20260812 03:05:02.053215 3903 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:02.054196 3903 heartbeater.cc:511] Master 127.10.118.126:46513 requested a full tablet report, sending...
I20260812 03:05:02.056486 3306 ts_manager.cc:194] Registered new tserver with Master: 4bc96d09a3a249f598ca4ba0c31df623 (127.10.118.66:46777)
I20260812 03:05:02.057928 3306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.66:55345
W20260812 03:05:02.361505 3913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:02.362066 3913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:02.362671 3913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:02.393402 3913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:02.393749 3913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:02.394706 3913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:05:02.429286 3913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:45331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:46513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:02.430629 3913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:02.432356 3913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:02.452687 3931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:03.061857 3903 heartbeater.cc:503] Master 127.10.118.126:46513 was elected leader, sending a full tablet report...
W20260812 03:05:02.452693 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:02.455466 3913 server_base.cc:1034] running on GCE node
W20260812 03:05:02.453665 3930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:03.646529 3913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:03.649132 3913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:03.650527 3913 hybrid_clock.cc:648] HybridClock initialized: now 1786503903650499 us; error 52 us; skew 500 ppm
I20260812 03:05:03.651356 3913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:05:03.662676 3913 webserver.cc:460] Webserver started at http://127.10.118.67:36333/ using document root <none> and password file <none>
I20260812 03:05:03.663645 3913 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:03.663888 3913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:03.664446 3913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:03.669878 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "81dab6bc6faa411bbb9099e7a86314b1"
format_stamp: "Formatted at 2026-08-12 03:05:03 on dist-test-slave-zpvx"
server_key: "821056e65d66c06a59029b611f69b537"
server_key_iv: "c54d16fd06c653d8e18c0311e9ca3941"
server_key_version: "encryptionkey@0"
I20260812 03:05:03.671093 3913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "81dab6bc6faa411bbb9099e7a86314b1"
format_stamp: "Formatted at 2026-08-12 03:05:03 on dist-test-slave-zpvx"
server_key: "821056e65d66c06a59029b611f69b537"
server_key_iv: "c54d16fd06c653d8e18c0311e9ca3941"
server_key_version: "encryptionkey@0"
I20260812 03:05:03.678356 3913 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.007s
I20260812 03:05:03.684011 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:03.685184 3913 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:05:03.685528 3913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "81dab6bc6faa411bbb9099e7a86314b1"
format_stamp: "Formatted at 2026-08-12 03:05:03 on dist-test-slave-zpvx"
server_key: "821056e65d66c06a59029b611f69b537"
server_key_iv: "c54d16fd06c653d8e18c0311e9ca3941"
server_key_version: "encryptionkey@0"
I20260812 03:05:03.685936 3913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:03.723531 3913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:03.725092 3913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:03.725567 3913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:03.728116 3913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:03.732456 3913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:03.732697 3913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:03.732971 3913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:03.733147 3913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:03.870675 3913 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:35535
I20260812 03:05:03.870800 4138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:35535 every 8 connection(s)
I20260812 03:05:03.873267 3913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:05:03.881232 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 3913
I20260812 03:05:03.881964 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:05:03.883132 10713 external_mini_cluster.cc:1428] Setting key a83a7ccc774cea407328b14b35439f1d
I20260812 03:05:03.896811 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:03.877698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:33171 (local address 127.10.118.126:46513)
0812 03:05:03.878678 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:03.878697 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:03.880825 (+ 2128us) server_negotiation.cc:407] Connection header received
0812 03:05:03.885635 (+ 4810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:03.885666 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:03.885891 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:03.886080 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:03.887916 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:03.889060 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:03.891158 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:03.891787 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:03.892070 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:03.893945 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:03.893965 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:03.893981 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:03.894026 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:03.894088 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:03.894122 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:03.894264 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:03.894602 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 03:05:03.894859 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":413,"threads_started":1}
I20260812 03:05:03.897514 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:03.878110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:46513 (local address 127.10.118.67:33171)
0812 03:05:03.880586 (+ 2476us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:03.880657 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:03.885327 (+ 4670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:03.886300 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:03.886323 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:03.886513 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:03.887634 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:03.887678 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:03.889328 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:03.889348 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:03.890898 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:03.890935 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:03.892113 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:03.892151 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:03.892658 (+ 507us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:03.893452 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:05:03.893470 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:03.893531 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:05:03.893639 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:03.894292 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:03.894340 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:03.894360 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 03:05:03.894540 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 03:05:03.894990 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1829,"thread_start_us":1132,"threads_started":1}
I20260812 03:05:03.899328 4139 heartbeater.cc:347] Connected to a master server at 127.10.118.126:46513
I20260812 03:05:03.899777 4139 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:03.900997 4139 heartbeater.cc:511] Master 127.10.118.126:46513 requested a full tablet report, sending...
I20260812 03:05:03.903298 3306 ts_manager.cc:194] Registered new tserver with Master: 81dab6bc6faa411bbb9099e7a86314b1 (127.10.118.67:35535)
I20260812 03:05:03.904759 3306 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.67:33171
I20260812 03:05:03.916316 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:05:03.942615 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:03.931799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.10.118.126:46513)
0812 03:05:03.932195 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:03.932216 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:03.933381 (+ 1165us) server_negotiation.cc:407] Connection header received
0812 03:05:03.933716 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:03.933733 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:03.933918 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:03.934053 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:03.935053 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:03.935959 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:03.938001 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:03.938579 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:03.938773 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:03.940470 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:03.940528 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:03.940571 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:03.940629 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:03.940704 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:03.940739 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:03.940930 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:03.941266 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 03:05:03.941404 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 03:05:03.949029 3306 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:05:03.970033 3306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:05:04.017386 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:03.996584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.10.118.67:35535)
0812 03:05:03.999256 (+ 2672us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:03.999288 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:03.999329 (+ 41us) server_negotiation.cc:407] Connection header received
0812 03:05:03.999495 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:03.999529 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:03.999932 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:04.000159 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:04.002567 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.004003 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.008601 (+ 4598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.011625 (+ 3024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.012165 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.013837 (+ 1672us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:04.015317 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0812 03:05:04.015677 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2148,"thread_start_us":1654,"threads_started":1}
I20260812 03:05:04.019253 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:03.996077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:35535 (local address 127.0.0.1:34090)
0812 03:05:03.998551 (+ 2474us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:03.998639 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:03.999008 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:04.000907 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:04.000946 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:04.001119 (+ 173us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:04.002173 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.002231 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.004336 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.004372 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.008135 (+ 3763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.008215 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.012762 (+ 4547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.012808 (+ 46us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.014712 (+ 1904us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.014961 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 03:05:04.015178 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 03:05:04.015572 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1945,"mutex_wait_us":1197,"thread_start_us":2410,"threads_started":1}
I20260812 03:05:04.023939 4029 tablet_service.cc:1467] Processing CreateTablet for tablet f80732d252c446438b6db65569b422cf (DEFAULT_TABLE table=test-table [id=042984840308414f887fa6b5c7787228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:04.026153 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80732d252c446438b6db65569b422cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:04.032166 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.000179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.10.118.65:38523)
0812 03:05:04.003136 (+ 2957us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:04.003178 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:04.008454 (+ 5276us) server_negotiation.cc:407] Connection header received
0812 03:05:04.008728 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:04.008769 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:04.009265 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:04.009543 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:04.016132 (+ 6589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.018252 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.023458 (+ 5206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.025612 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.025940 (+ 328us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.027237 (+ 1297us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:04.030053 (+ 2816us) server_negotiation.cc:299] Negotiation successful
0812 03:05:04.030377 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2290,"thread_start_us":1858,"threads_started":1}
I20260812 03:05:04.034250 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.004325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:46777 (local address 127.0.0.1:57566)
0812 03:05:04.016195 (+ 11870us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:04.016265 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:04.016540 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:04.018316 (+ 1776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:04.018426 (+ 110us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:04.018521 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:04.021138 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.021193 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.023838 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.023878 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.026904 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.026959 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.030343 (+ 3384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.030374 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.031302 (+ 928us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.031447 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 03:05:04.031592 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 03:05:04.031904 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11432,"mutex_wait_us":272,"thread_start_us":10620,"threads_started":1}
I20260812 03:05:04.036549 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.004845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.10.118.66:46777)
0812 03:05:04.016854 (+ 12009us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:04.016888 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:04.016946 (+ 58us) server_negotiation.cc:407] Connection header received
0812 03:05:04.017128 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:04.017159 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:04.017556 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:04.017776 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:04.021502 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.023510 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.027259 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.029304 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.029595 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.030111 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:04.034743 (+ 4632us) server_negotiation.cc:299] Negotiation successful
0812 03:05:04.035045 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9882,"thread_start_us":9566,"threads_started":1}
I20260812 03:05:04.034739 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:03.999717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:38523 (local address 127.0.0.1:59014)
0812 03:05:04.007716 (+ 7999us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:04.007799 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:04.008140 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:04.010362 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:04.010403 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:04.010539 (+ 136us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:04.015707 (+ 5168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.015762 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.018582 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.019070 (+ 488us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.023119 (+ 4049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.023171 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.026848 (+ 3677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.026881 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.029269 (+ 2388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.029423 (+ 154us) client_negotiation.cc:769] Sending connection context
0812 03:05:04.029560 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 03:05:04.032810 (+ 3250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4087,"mutex_wait_us":3443,"thread_start_us":3831,"threads_started":1}
I20260812 03:05:04.041958 3783 tablet_service.cc:1467] Processing CreateTablet for tablet f80732d252c446438b6db65569b422cf (DEFAULT_TABLE table=test-table [id=042984840308414f887fa6b5c7787228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:04.042541 3532 tablet_service.cc:1467] Processing CreateTablet for tablet f80732d252c446438b6db65569b422cf (DEFAULT_TABLE table=test-table [id=042984840308414f887fa6b5c7787228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:04.043569 3783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80732d252c446438b6db65569b422cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:04.044454 3532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f80732d252c446438b6db65569b422cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:04.058662 4173 tablet_bootstrap.cc:492] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Bootstrap starting.
I20260812 03:05:04.066205 4173 tablet_bootstrap.cc:654] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:04.069345 4173 log.cc:824] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:04.072347 4174 tablet_bootstrap.cc:492] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Bootstrap starting.
I20260812 03:05:04.081872 4174 tablet_bootstrap.cc:654] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:04.084946 4174 log.cc:824] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:04.086638 4173 tablet_bootstrap.cc:492] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: No bootstrap required, opened a new log
I20260812 03:05:04.087158 4173 ts_tablet_manager.cc:1397] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.008s
I20260812 03:05:04.087234 4178 tablet_bootstrap.cc:492] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442: Bootstrap starting.
I20260812 03:05:04.094659 4178 tablet_bootstrap.cc:654] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:04.097574 4178 log.cc:824] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:04.099195 4174 tablet_bootstrap.cc:492] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: No bootstrap required, opened a new log
I20260812 03:05:04.099766 4174 ts_tablet_manager.cc:1397] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Time spent bootstrapping tablet: real 0.028s user 0.020s sys 0.004s
I20260812 03:05:04.113093 4178 tablet_bootstrap.cc:492] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442: No bootstrap required, opened a new log
I20260812 03:05:04.113685 4178 ts_tablet_manager.cc:1397] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.013s
I20260812 03:05:04.118942 4173 raft_consensus.cc:348] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.119836 4173 raft_consensus.cc:374] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:04.120119 4173 raft_consensus.cc:729] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dab6bc6faa411bbb9099e7a86314b1, State: Initialized, Role: FOLLOWER
I20260812 03:05:04.121016 4173 consensus_queue.cc:260] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.132413 4174 raft_consensus.cc:348] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.133483 4174 raft_consensus.cc:374] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:04.133853 4174 raft_consensus.cc:729] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bc96d09a3a249f598ca4ba0c31df623, State: Initialized, Role: FOLLOWER
I20260812 03:05:04.134868 4174 consensus_queue.cc:260] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.136081 4139 heartbeater.cc:503] Master 127.10.118.126:46513 was elected leader, sending a full tablet report...
I20260812 03:05:04.136837 4173 ts_tablet_manager.cc:1428] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Time spent starting tablet: real 0.049s user 0.035s sys 0.015s
I20260812 03:05:04.148746 4178 raft_consensus.cc:348] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.149639 4178 raft_consensus.cc:374] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:04.149932 4178 raft_consensus.cc:729] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa99fe9561684fcc9933d36590057442, State: Initialized, Role: FOLLOWER
I20260812 03:05:04.152232 4178 consensus_queue.cc:260] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.153636 4174 ts_tablet_manager.cc:1428] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Time spent starting tablet: real 0.054s user 0.028s sys 0.022s
I20260812 03:05:04.156579 4178 ts_tablet_manager.cc:1428] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442: Time spent starting tablet: real 0.043s user 0.036s sys 0.003s
W20260812 03:05:04.188419 3657 tablet.cc:2369] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:05:04.239415 4184 raft_consensus.cc:482] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:05:04.239887 4184 raft_consensus.cc:504] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.242316 4184 leader_election.cc:288] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81dab6bc6faa411bbb9099e7a86314b1 (127.10.118.67:35535), fa99fe9561684fcc9933d36590057442 (127.10.118.65:38523)
I20260812 03:05:04.256007 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.242978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:39621 (local address 127.10.118.67:35535)
0812 03:05:04.243584 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:04.243619 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:04.244908 (+ 1289us) server_negotiation.cc:407] Connection header received
0812 03:05:04.245440 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:04.245483 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:04.245851 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:04.246055 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:04.247390 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.248821 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.252024 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.253685 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.254046 (+ 361us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.254216 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:04.254869 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 03:05:04.255057 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260812 03:05:04.256971 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.242723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:35535 (local address 127.10.118.66:39621)
0812 03:05:04.244717 (+ 1994us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:04.244771 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:04.245148 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:04.246177 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:04.246216 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:04.246359 (+ 143us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:04.247086 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.247123 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.249077 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.249098 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.251744 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.251784 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.254052 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.254086 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.254491 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.254630 (+ 139us) client_negotiation.cc:769] Sending connection context
0812 03:05:04.254800 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 03:05:04.255079 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"mutex_wait_us":73,"thread_start_us":1268,"threads_started":1}
I20260812 03:05:04.258877 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80732d252c446438b6db65569b422cf" candidate_uuid: "4bc96d09a3a249f598ca4ba0c31df623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dab6bc6faa411bbb9099e7a86314b1" is_pre_election: true
I20260812 03:05:04.259354 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.245175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:34371 (local address 127.10.118.65:38523)
0812 03:05:04.245818 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:04.245851 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:04.247284 (+ 1433us) server_negotiation.cc:407] Connection header received
0812 03:05:04.247654 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:04.247673 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:04.247913 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:04.248032 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:04.249461 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.250800 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.253967 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.255650 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.256757 (+ 1107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.257016 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:04.257679 (+ 663us) server_negotiation.cc:299] Negotiation successful
0812 03:05:04.257959 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260812 03:05:04.259819 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.244867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:38523 (local address 127.10.118.66:34371)
0812 03:05:04.247049 (+ 2182us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:04.247132 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:04.247410 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:04.248132 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:04.248160 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:04.248258 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:04.249189 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.249228 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.251032 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.251053 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.253667 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:04.253715 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.256816 (+ 3101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.256852 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:04.257308 (+ 456us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.257438 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 03:05:04.257617 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 03:05:04.257911 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1821,"mutex_wait_us":81,"thread_start_us":1431,"threads_started":1}
I20260812 03:05:04.259889 4062 raft_consensus.cc:2454] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc96d09a3a249f598ca4ba0c31df623 in term 0.
I20260812 03:05:04.261200 3699 leader_election.cc:302] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc96d09a3a249f598ca4ba0c31df623, 81dab6bc6faa411bbb9099e7a86314b1; no voters:
I20260812 03:05:04.261353 3571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80732d252c446438b6db65569b422cf" candidate_uuid: "4bc96d09a3a249f598ca4ba0c31df623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa99fe9561684fcc9933d36590057442" is_pre_election: true
I20260812 03:05:04.261977 4184 raft_consensus.cc:2781] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:05:04.262128 3571 raft_consensus.cc:2454] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bc96d09a3a249f598ca4ba0c31df623 in term 0.
I20260812 03:05:04.262260 4184 raft_consensus.cc:482] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:05:04.262531 4184 raft_consensus.cc:3037] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:04.267386 4184 raft_consensus.cc:504] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
I20260812 03:05:04.268950 4184 leader_election.cc:288] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [CANDIDATE]: Term 1 election: Requested vote from peers 81dab6bc6faa411bbb9099e7a86314b1 (127.10.118.67:35535), fa99fe9561684fcc9933d36590057442 (127.10.118.65:38523)
I20260812 03:05:04.269691 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80732d252c446438b6db65569b422cf" candidate_uuid: "4bc96d09a3a249f598ca4ba0c31df623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dab6bc6faa411bbb9099e7a86314b1"
I20260812 03:05:04.269929 3571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80732d252c446438b6db65569b422cf" candidate_uuid: "4bc96d09a3a249f598ca4ba0c31df623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa99fe9561684fcc9933d36590057442"
I20260812 03:05:04.270161 4062 raft_consensus.cc:3037] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:04.270414 3571 raft_consensus.cc:3037] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:04.274852 4062 raft_consensus.cc:2454] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc96d09a3a249f598ca4ba0c31df623 in term 1.
I20260812 03:05:04.275099 3571 raft_consensus.cc:2454] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bc96d09a3a249f598ca4ba0c31df623 in term 1.
I20260812 03:05:04.275899 3699 leader_election.cc:302] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bc96d09a3a249f598ca4ba0c31df623, 81dab6bc6faa411bbb9099e7a86314b1; no voters:
I20260812 03:05:04.276589 4184 raft_consensus.cc:2781] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:04.278230 4184 raft_consensus.cc:686] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 1 LEADER]: Becoming Leader. State: Replica: 4bc96d09a3a249f598ca4ba0c31df623, State: Running, Role: LEADER
I20260812 03:05:04.279011 4184 consensus_queue.cc:237] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } }
W20260812 03:05:04.281101 3910 tablet.cc:2369] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:05:04.291217 3306 catalog_manager.cc:5495] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bc96d09a3a249f598ca4ba0c31df623 (127.10.118.66). New cstate: current_term: 1 leader_uuid: "4bc96d09a3a249f598ca4ba0c31df623" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bc96d09a3a249f598ca4ba0c31df623" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 46777 } health_report { overall_health: HEALTHY } } }
I20260812 03:05:04.338444 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.328778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.10.118.66:46777)
0812 03:05:04.329414 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:04.329442 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:04.329494 (+ 52us) server_negotiation.cc:407] Connection header received
0812 03:05:04.329941 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:04.329968 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:04.330285 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:04.330468 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:05:04.331407 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.332270 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.334271 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:04.334835 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:04.335121 (+ 286us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:04.336140 (+ 1019us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:05:04.336653 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:05:04.336907 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:04.337294 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 03:05:04.337395 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260812 03:05:04.351516 3571 raft_consensus.cc:1261] T f80732d252c446438b6db65569b422cf P fa99fe9561684fcc9933d36590057442 [term 1 FOLLOWER]: Refusing update from remote peer 4bc96d09a3a249f598ca4ba0c31df623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:05:04.351679 4062 raft_consensus.cc:1261] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 1 FOLLOWER]: Refusing update from remote peer 4bc96d09a3a249f598ca4ba0c31df623: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:05:04.353142 4191 consensus_queue.cc:1035] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa99fe9561684fcc9933d36590057442" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 38523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260812 03:05:04.353878 4184 consensus_queue.cc:1035] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81dab6bc6faa411bbb9099e7a86314b1" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 35535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:05:04.374796 4196 mvcc.cc:205] Tried to move back new op lower bound from 7317519992207343616 to 7317519991934464000. Current Snapshot: MvccSnapshot[applied={T|T < 7317519992207343616}]
I20260812 03:05:04.408191 3306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:05:04.408821 3306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:05:04.418615 3306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8dd42023b5f24f118238bbff7e5dc691: Sending DeleteTablet for 3 replicas of tablet f80732d252c446438b6db65569b422cf
I20260812 03:05:04.420471 4029 tablet_service.cc:1514] Processing DeleteTablet for tablet f80732d252c446438b6db65569b422cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:04 UTC) from {username='slave'} at 127.0.0.1:34090
I20260812 03:05:04.421518 3783 tablet_service.cc:1514] Processing DeleteTablet for tablet f80732d252c446438b6db65569b422cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:04 UTC) from {username='slave'} at 127.0.0.1:57566
I20260812 03:05:04.422295 3532 tablet_service.cc:1514] Processing DeleteTablet for tablet f80732d252c446438b6db65569b422cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:04 UTC) from {username='slave'} at 127.0.0.1:59014
I20260812 03:05:04.424211 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 3421
I20260812 03:05:04.437525 4216 tablet_replica.cc:330] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: stopping tablet replica
I20260812 03:05:04.438509 4216 raft_consensus.cc:2229] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:05:04.439946 4216 raft_consensus.cc:2258] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:05:04.445159 4216 ts_tablet_manager.cc:1905] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:05:04.446116 4214 tablet_replica.cc:330] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: stopping tablet replica
I20260812 03:05:04.447119 4214 raft_consensus.cc:2229] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:05:04.448050 4214 raft_consensus.cc:2258] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:05:04.451570 4214 ts_tablet_manager.cc:1905] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:05:04.463299 4216 ts_tablet_manager.cc:1918] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:05:04.463742 4216 log.cc:1196] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/f80732d252c446438b6db65569b422cf
I20260812 03:05:04.464788 4216 ts_tablet_manager.cc:1939] T f80732d252c446438b6db65569b422cf P 4bc96d09a3a249f598ca4ba0c31df623: Deleting consensus metadata
I20260812 03:05:04.467156 4214 ts_tablet_manager.cc:1918] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:05:04.467657 4214 log.cc:1196] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/f80732d252c446438b6db65569b422cf
I20260812 03:05:04.468168 3282 catalog_manager.cc:4841] TS 4bc96d09a3a249f598ca4ba0c31df623 (127.10.118.66:46777): tablet f80732d252c446438b6db65569b422cf (table test-table [id=042984840308414f887fa6b5c7787228]) successfully deleted
I20260812 03:05:04.468559 4214 ts_tablet_manager.cc:1939] T f80732d252c446438b6db65569b422cf P 81dab6bc6faa411bbb9099e7a86314b1: Deleting consensus metadata
I20260812 03:05:04.471596 3282 catalog_manager.cc:4841] TS 81dab6bc6faa411bbb9099e7a86314b1 (127.10.118.67:35535): tablet f80732d252c446438b6db65569b422cf (table test-table [id=042984840308414f887fa6b5c7787228]) successfully deleted
W20260812 03:05:04.478063 3282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:38523 (error 108)
I20260812 03:05:04.479245 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 3662
I20260812 03:05:04.497083 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:04.495916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:38523 (local address 127.0.0.1:59030)
0812 03:05:04.496488 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:04.496679 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:38523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260812 03:05:04.497592 3282 catalog_manager.cc:4584] TS fa99fe9561684fcc9933d36590057442 (127.10.118.65:38523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f80732d252c446438b6db65569b422cf: Network error: Client connection negotiation failed: client connection to 127.10.118.65:38523: connect: Connection refused (error 111)
I20260812 03:05:04.523988 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 3913
I20260812 03:05:04.552502 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 3247
2026-08-12T03:05:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (8323 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:04 dist-test-slave-zpvx krb5kdc[4228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:04 dist-test-slave-zpvx krb5kdc[4228](info): set up 2 sockets
Aug 12 03:05:04 dist-test-slave-zpvx krb5kdc[4228](info): commencing operation
krb5kdc: starting...
W20260812 03:05:05.961532 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:06 dist-test-slave-zpvx krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:05:06.201182 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38933
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:38933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:06.522341 4257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:06.522979 4257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:06.523610 4257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:06.553974 4257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:06.554368 4257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:06.554697 4257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:06.554963 4257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:06.555227 4257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:06.591538 4257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:38933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:06.592870 4257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:06.593988 4257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:06.613061 4274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:06.613061 4269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:06.613054 4270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:06.614295 4257 server_base.cc:1034] running on GCE node
I20260812 03:05:07.877408 4257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:07.880194 4257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:07.881604 4257 hybrid_clock.cc:648] HybridClock initialized: now 1786503907881577 us; error 51 us; skew 500 ppm
I20260812 03:05:07.882469 4257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:07 dist-test-slave-zpvx krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503907, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:07.888629 4257 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:05:07.893993 4257 webserver.cc:460] Webserver started at http://127.10.118.126:45267/ using document root <none> and password file <none>
I20260812 03:05:07.894891 4257 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:07.895072 4257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:07.895484 4257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:07.900857 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "5bf7cfe8968d48e996e796a8f60114a3"
format_stamp: "Formatted at 2026-08-12 03:05:07 on dist-test-slave-zpvx"
server_key: "01afe8eeabed379f2f28a7e8ff40037b"
server_key_iv: "579ac0f5bb4a19efe0681219a33519d4"
server_key_version: "encryptionkey@0"
I20260812 03:05:07.901989 4257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "5bf7cfe8968d48e996e796a8f60114a3"
format_stamp: "Formatted at 2026-08-12 03:05:07 on dist-test-slave-zpvx"
server_key: "01afe8eeabed379f2f28a7e8ff40037b"
server_key_iv: "579ac0f5bb4a19efe0681219a33519d4"
server_key_version: "encryptionkey@0"
I20260812 03:05:07.909284 4257 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 03:05:07.914803 4286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:07.915891 4257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:05:07.916225 4257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "5bf7cfe8968d48e996e796a8f60114a3"
format_stamp: "Formatted at 2026-08-12 03:05:07 on dist-test-slave-zpvx"
server_key: "01afe8eeabed379f2f28a7e8ff40037b"
server_key_iv: "579ac0f5bb4a19efe0681219a33519d4"
server_key_version: "encryptionkey@0"
I20260812 03:05:07.916659 4257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:07.955039 4257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:07.960712 4257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:07.961222 4257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:08.030308 4257 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:38933
I20260812 03:05:08.030392 4386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:38933 every 8 connection(s)
I20260812 03:05:08.033167 4257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:08.038729 4387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:08.041337 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 4257
I20260812 03:05:08.041798 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:08.042939 10713 external_mini_cluster.cc:1428] Setting key 2b85c2c481c71db505028dc2d56a2951
I20260812 03:05:08.060017 4387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3: Bootstrap starting.
I20260812 03:05:08.066934 4387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3: Neither blocks nor log segments found. Creating new log.
Aug 12 03:05:08 dist-test-slave-zpvx krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:05:08.068900 4387 log.cc:824] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:08.075258 4387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3: No bootstrap required, opened a new log
I20260812 03:05:08.083278 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:08.045428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.10.118.126:38933)
0812 03:05:08.048201 (+ 2773us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:08.048228 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:08.048330 (+ 102us) server_negotiation.cc:407] Connection header received
0812 03:05:08.052275 (+ 3945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:08.052324 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:08.052731 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:08.053066 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:08.056600 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:08.058226 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:08.061018 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:08.061829 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:08.062054 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:08.069083 (+ 7029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:08.069131 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:08.069177 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:08.069334 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:08.075326 (+ 5992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:08.076529 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:08.076565 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:08.076592 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:08.076902 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:08.077554 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:08.077582 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:08.077601 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:08.078576 (+ 975us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:08.079049 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:08.079632 (+ 583us) server_negotiation.cc:299] Negotiation successful
0812 03:05:08.080286 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2347,"thread_start_us":1823,"threads_started":1}
I20260812 03:05:08.099135 4387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf7cfe8968d48e996e796a8f60114a3" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38933 } }
I20260812 03:05:08.099800 4387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:08.100004 4387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf7cfe8968d48e996e796a8f60114a3, State: Initialized, Role: FOLLOWER
I20260812 03:05:08.100639 4387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf7cfe8968d48e996e796a8f60114a3" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38933 } }
I20260812 03:05:08.101120 4387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:08.101341 4387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:08.101588 4387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:08.105867 4387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf7cfe8968d48e996e796a8f60114a3" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38933 } }
I20260812 03:05:08.106590 4387 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf7cfe8968d48e996e796a8f60114a3; no voters:
I20260812 03:05:08.108436 4387 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:08.108824 4395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:08.111452 4395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf7cfe8968d48e996e796a8f60114a3, State: Running, Role: LEADER
I20260812 03:05:08.112254 4395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf7cfe8968d48e996e796a8f60114a3" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38933 } }
I20260812 03:05:08.113469 4387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:08.124274 4398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf7cfe8968d48e996e796a8f60114a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf7cfe8968d48e996e796a8f60114a3" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38933 } } }
I20260812 03:05:08.125994 4398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:08.125339 4401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf7cfe8968d48e996e796a8f60114a3. Latest consensus state: current_term: 1 leader_uuid: "5bf7cfe8968d48e996e796a8f60114a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf7cfe8968d48e996e796a8f60114a3" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38933 } } }
I20260812 03:05:08.127679 4401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:08.129238 4416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:08.139279 4416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:08.157021 4416 catalog_manager.cc:1347] Generated new cluster ID: 6e93591b14184087ae90294e3b057f83
I20260812 03:05:08.157377 4416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:08.185400 4416 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:08.187961 4416 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:08.201979 4416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3: Generated new TSK 0
I20260812 03:05:08.203115 4416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.118.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
Entry for principal kudu/127.10.118.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab.
I20260812 03:05:08.279058 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38933
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:08.591799 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:08.592324 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:08.593020 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:08.625080 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:08.625476 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:08.626452 4443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:05:08.662243 4443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.65.keytab
--principal=kudu/127.10.118.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:08.663441 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:08.664565 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:08.695510 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:08.698426 4455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:10.075064 4443 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260812 03:05:10.075654 4454 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1380 milliseconds
W20260812 03:05:08.697533 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:10.080446 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:10.083307 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:10.084777 4443 hybrid_clock.cc:648] HybridClock initialized: now 1786503910084741 us; error 39 us; skew 500 ppm
I20260812 03:05:10.085903 4443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:10 dist-test-slave-zpvx krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503910, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:10.093183 4443 init.cc:377] Logged in from keytab as kudu/127.10.118.65@KRBTEST.COM (short username kudu)
I20260812 03:05:10.099073 4443 webserver.cc:460] Webserver started at http://127.10.118.65:38333/ using document root <none> and password file <none>
I20260812 03:05:10.100000 4443 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:10.100235 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:10.100732 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:10.105883 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "67c8358c5491423b8021429267bd9a48"
format_stamp: "Formatted at 2026-08-12 03:05:10 on dist-test-slave-zpvx"
server_key: "46815958a2a388ed72c67228b7b7c78f"
server_key_iv: "34249a380d89209bb3f4cbcb6b2a1aa4"
server_key_version: "encryptionkey@0"
I20260812 03:05:10.107033 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "67c8358c5491423b8021429267bd9a48"
format_stamp: "Formatted at 2026-08-12 03:05:10 on dist-test-slave-zpvx"
server_key: "46815958a2a388ed72c67228b7b7c78f"
server_key_iv: "34249a380d89209bb3f4cbcb6b2a1aa4"
server_key_version: "encryptionkey@0"
I20260812 03:05:10.114950 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260812 03:05:10.121167 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:10.122342 4443 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 03:05:10.122660 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "67c8358c5491423b8021429267bd9a48"
format_stamp: "Formatted at 2026-08-12 03:05:10 on dist-test-slave-zpvx"
server_key: "46815958a2a388ed72c67228b7b7c78f"
server_key_iv: "34249a380d89209bb3f4cbcb6b2a1aa4"
server_key_version: "encryptionkey@0"
I20260812 03:05:10.123018 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:10.169081 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:10.174938 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:10.175421 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:10.178745 4443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:10.183549 4443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:10.183909 4443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 03:05:10.184266 4443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:10.184470 4443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 03:05:10.372951 4443 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:43301
I20260812 03:05:10.373122 4677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:43301 every 8 connection(s)
I20260812 03:05:10.375808 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:05:10.381070 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 4443
I20260812 03:05:10.381592 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:05:10.382946 10713 external_mini_cluster.cc:1428] Setting key 6cab73728889a2c758ec58029d9deda5
Aug 12 03:05:10 dist-test-slave-zpvx krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503910, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.65@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.66@KRBTEST.COM" created.
I20260812 03:05:10.429927 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:10.384606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:53279 (local address 127.10.118.126:38933)
0812 03:05:10.401548 (+ 16942us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:10.401574 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:10.401633 (+ 59us) server_negotiation.cc:407] Connection header received
0812 03:05:10.404087 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:10.404116 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:10.404339 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:10.404556 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:10.407496 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:10.408819 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:10.411244 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:10.411885 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:10.412174 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:10.417486 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:10.417507 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:10.417529 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:10.417627 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:10.421793 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:10.423305 (+ 1512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:10.423327 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:10.423341 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:10.423550 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:10.424458 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:10.424487 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:10.424507 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:10.425179 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:10.425411 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:10.427036 (+ 1625us) server_negotiation.cc:299] Negotiation successful
0812 03:05:10.427538 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16504,"thread_start_us":16959,"threads_started":1}
I20260812 03:05:10.430702 4683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:10.383652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38933 (local address 127.10.118.65:53279)
0812 03:05:10.396281 (+ 12629us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:10.396410 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:10.403705 (+ 7295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:10.404896 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:10.404947 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:10.406208 (+ 1261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:10.407185 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:10.407233 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:10.409107 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:10.409133 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:10.410960 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:10.410999 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:10.412255 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:10.412287 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:10.412771 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:10.413642 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:05:10.413708 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:10.417131 (+ 3423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:10.422041 (+ 4910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:05:10.422075 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:05:10.422109 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:05:10.423076 (+ 967us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:05:10.423819 (+ 743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:05:10.423846 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:05:10.423872 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:05:10.424210 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:05:10.425442 (+ 1232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:10.425490 (+ 48us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:10.426182 (+ 692us) client_negotiation.cc:769] Sending connection context
0812 03:05:10.426663 (+ 481us) client_negotiation.cc:240] Negotiation successful
0812 03:05:10.427512 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11871,"thread_start_us":16543,"threads_started":1}
I20260812 03:05:10.433131 4680 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38933
I20260812 03:05:10.433626 4680 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:10.434787 4680 heartbeater.cc:511] Master 127.10.118.126:38933 requested a full tablet report, sending...
I20260812 03:05:10.438262 4315 ts_manager.cc:194] Registered new tserver with Master: 67c8358c5491423b8021429267bd9a48 (127.10.118.65:43301)
I20260812 03:05:10.441663 4315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.65@KRBTEST.COM'} at 127.10.118.65:53279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
Entry for principal kudu/127.10.118.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab.
I20260812 03:05:10.458806 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38933
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:10.774165 4694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:10.774727 4694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:10.775388 4694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:10.805747 4694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:10.806149 4694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:10.807092 4694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:05:10.843179 4694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.66.keytab
--principal=kudu/127.10.118.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:10.844544 4694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:10.845582 4694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:10.863904 4709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:11.447000 4680 heartbeater.cc:503] Master 127.10.118.126:38933 was elected leader, sending a full tablet report...
W20260812 03:05:10.863930 4708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:10.866323 4694 server_base.cc:1034] running on GCE node
W20260812 03:05:10.864197 4713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:12.038311 4694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:12.041172 4694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:12.042613 4694 hybrid_clock.cc:648] HybridClock initialized: now 1786503912042549 us; error 80 us; skew 500 ppm
I20260812 03:05:12.043560 4694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:12 dist-test-slave-zpvx krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503912, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:12.050101 4694 init.cc:377] Logged in from keytab as kudu/127.10.118.66@KRBTEST.COM (short username kudu)
I20260812 03:05:12.055150 4694 webserver.cc:460] Webserver started at http://127.10.118.66:35737/ using document root <none> and password file <none>
I20260812 03:05:12.056111 4694 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:12.056313 4694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:12.056782 4694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:12.062538 4694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "05e3b3fc78784a15a900dde5612efcf4"
format_stamp: "Formatted at 2026-08-12 03:05:12 on dist-test-slave-zpvx"
server_key: "e28ee5c0ac3cf1c3b7d034afcbeba701"
server_key_iv: "00af77dc22ec711a69d2978adcea06eb"
server_key_version: "encryptionkey@0"
I20260812 03:05:12.063903 4694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "05e3b3fc78784a15a900dde5612efcf4"
format_stamp: "Formatted at 2026-08-12 03:05:12 on dist-test-slave-zpvx"
server_key: "e28ee5c0ac3cf1c3b7d034afcbeba701"
server_key_iv: "00af77dc22ec711a69d2978adcea06eb"
server_key_version: "encryptionkey@0"
I20260812 03:05:12.071704 4694 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:05:12.077796 4726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:12.078879 4694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:05:12.079252 4694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "05e3b3fc78784a15a900dde5612efcf4"
format_stamp: "Formatted at 2026-08-12 03:05:12 on dist-test-slave-zpvx"
server_key: "e28ee5c0ac3cf1c3b7d034afcbeba701"
server_key_iv: "00af77dc22ec711a69d2978adcea06eb"
server_key_version: "encryptionkey@0"
I20260812 03:05:12.079684 4694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:12.127841 4694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:12.133673 4694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:12.134141 4694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:12.136879 4694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:12.140880 4694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:12.141104 4694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:12.141371 4694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:12.141542 4694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:12.280179 4694 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:39591
I20260812 03:05:12.280304 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:39591 every 8 connection(s)
I20260812 03:05:12.282997 4694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:05:12.288897 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 4694
I20260812 03:05:12.289511 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:05:12.290704 10713 external_mini_cluster.cc:1428] Setting key c8a4cfea8616dbe99dfa1e85e1c18d2b
Aug 12 03:05:12 dist-test-slave-zpvx krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503912, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.66@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.118.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.67@KRBTEST.COM" created.
I20260812 03:05:12.323185 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:12.287594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:55657 (local address 127.10.118.126:38933)
0812 03:05:12.289275 (+ 1681us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:12.289301 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:12.291297 (+ 1996us) server_negotiation.cc:407] Connection header received
0812 03:05:12.296713 (+ 5416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:12.296751 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:12.297038 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:12.297267 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:12.299446 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:12.301184 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:12.303507 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:12.304751 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:12.305086 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:12.310758 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:12.310802 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:12.310838 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:12.311000 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:12.316617 (+ 5617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:12.317824 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:12.317856 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:12.317879 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:12.318160 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:12.319041 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:12.319070 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:12.319103 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:12.319828 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:12.320108 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:12.321225 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0812 03:05:12.321565 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":692,"threads_started":1}
I20260812 03:05:12.324395 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:12.287818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38933 (local address 127.10.118.66:55657)
0812 03:05:12.291045 (+ 3227us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:12.291115 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:12.296294 (+ 5179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:12.297471 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:12.297494 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:12.298291 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:12.299143 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:12.299186 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:12.301443 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:12.301468 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:12.303238 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:12.303276 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:12.305149 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:12.305192 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:12.305806 (+ 614us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:12.306955 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:05:12.307021 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:12.310401 (+ 3380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:12.316839 (+ 6438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:05:12.316863 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:05:12.316890 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:05:12.317621 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:05:12.318459 (+ 838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:05:12.318480 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:05:12.318494 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:05:12.318787 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:05:12.320087 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:12.320122 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:12.320631 (+ 509us) client_negotiation.cc:769] Sending connection context
0812 03:05:12.321038 (+ 407us) client_negotiation.cc:240] Negotiation successful
0812 03:05:12.321549 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2645,"thread_start_us":1752,"threads_started":1}
I20260812 03:05:12.326411 4933 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38933
I20260812 03:05:12.326839 4933 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:12.327781 4933 heartbeater.cc:511] Master 127.10.118.126:38933 requested a full tablet report, sending...
I20260812 03:05:12.330170 4315 ts_manager.cc:194] Registered new tserver with Master: 05e3b3fc78784a15a900dde5612efcf4 (127.10.118.66:39591)
I20260812 03:05:12.331717 4315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.66@KRBTEST.COM'} at 127.10.118.66:55657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
Entry for principal kudu/127.10.118.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab.
I20260812 03:05:12.355088 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38933
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:12.673768 4946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:12.674244 4946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:12.674896 4946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:12.708598 4946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:12.708914 4946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:12.709821 4946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:05:12.745234 4946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.67.keytab
--principal=kudu/127.10.118.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:38933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:12.746332 4946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:12.747362 4946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:12.765673 4962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:13.336624 4933 heartbeater.cc:503] Master 127.10.118.126:38933 was elected leader, sending a full tablet report...
W20260812 03:05:12.765745 4967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:12.765679 4963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:13.962978 4964 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1195 milliseconds
I20260812 03:05:13.963107 4946 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:05:13.964512 4946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:13.967324 4946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:13.968772 4946 hybrid_clock.cc:648] HybridClock initialized: now 1786503913968716 us; error 75 us; skew 500 ppm
I20260812 03:05:13.969573 4946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:13 dist-test-slave-zpvx krb5kdc[4228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503913, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:13.975713 4946 init.cc:377] Logged in from keytab as kudu/127.10.118.67@KRBTEST.COM (short username kudu)
I20260812 03:05:13.981015 4946 webserver.cc:460] Webserver started at http://127.10.118.67:37589/ using document root <none> and password file <none>
I20260812 03:05:13.981956 4946 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:13.982162 4946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:13.982654 4946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:13.987844 4946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "e66edf8efbec4684bd88334d0d2d3943"
format_stamp: "Formatted at 2026-08-12 03:05:13 on dist-test-slave-zpvx"
server_key: "cd59fdce07c05cfe8d72e3e6eca59298"
server_key_iv: "163aef34359b621267ddd59d64587465"
server_key_version: "encryptionkey@0"
I20260812 03:05:13.989077 4946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "e66edf8efbec4684bd88334d0d2d3943"
format_stamp: "Formatted at 2026-08-12 03:05:13 on dist-test-slave-zpvx"
server_key: "cd59fdce07c05cfe8d72e3e6eca59298"
server_key_iv: "163aef34359b621267ddd59d64587465"
server_key_version: "encryptionkey@0"
I20260812 03:05:13.996515 4946 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:05:14.002401 4982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:14.003543 4946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:05:14.003903 4946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "e66edf8efbec4684bd88334d0d2d3943"
format_stamp: "Formatted at 2026-08-12 03:05:13 on dist-test-slave-zpvx"
server_key: "cd59fdce07c05cfe8d72e3e6eca59298"
server_key_iv: "163aef34359b621267ddd59d64587465"
server_key_version: "encryptionkey@0"
I20260812 03:05:14.004309 4946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:14.041755 4946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:14.047696 4946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:14.048202 4946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:14.050714 4946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:14.054780 4946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:14.055004 4946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:14.055276 4946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:14.055449 4946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:14.193076 4946 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:42043
I20260812 03:05:14.193202 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:42043 every 8 connection(s)
I20260812 03:05:14.195820 4946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:05:14.205452 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 4946
I20260812 03:05:14.206175 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:05:14.207384 10713 external_mini_cluster.cc:1428] Setting key e773d7e42dea76d4a758c9ccc68fb8b2
Aug 12 03:05:14 dist-test-slave-zpvx krb5kdc[4228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503913, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.67@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:05:14.234390 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.200213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:44177 (local address 127.10.118.126:38933)
0812 03:05:14.205412 (+ 5199us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.205433 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.205489 (+ 56us) server_negotiation.cc:407] Connection header received
0812 03:05:14.209644 (+ 4155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.209665 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.209834 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.210031 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:14.213135 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.214240 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.216518 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.217142 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.217337 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.222683 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:14.222709 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:14.222725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:14.222824 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:14.228271 (+ 5447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:14.229564 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:14.229605 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:14.229631 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:14.229896 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:14.230653 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:14.230688 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:14.230716 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:14.231333 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:14.231534 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.232626 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.232992 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4719,"thread_start_us":4360,"threads_started":1}
I20260812 03:05:14.234985 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.200537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:38933 (local address 127.10.118.67:44177)
0812 03:05:14.203866 (+ 3329us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.203934 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:14.209291 (+ 5357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:14.210363 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:14.210388 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:14.211419 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:14.212719 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.212784 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.214474 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.214496 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.216239 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.216278 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.217452 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.217480 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.217855 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.218781 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:05:14.218853 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:14.222338 (+ 3485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:14.228540 (+ 6202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:05:14.228565 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:05:14.228592 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:05:14.229336 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:05:14.230095 (+ 759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:05:14.230114 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:05:14.230127 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:05:14.230397 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:05:14.231532 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:14.231567 (+ 35us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:14.231984 (+ 417us) client_negotiation.cc:769] Sending connection context
0812 03:05:14.232326 (+ 342us) client_negotiation.cc:240] Negotiation successful
0812 03:05:14.232889 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2761,"thread_start_us":1935,"threads_started":1}
I20260812 03:05:14.236927 5201 heartbeater.cc:347] Connected to a master server at 127.10.118.126:38933
I20260812 03:05:14.237342 5201 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:14.238325 5201 heartbeater.cc:511] Master 127.10.118.126:38933 requested a full tablet report, sending...
I20260812 03:05:14.241036 4315 ts_manager.cc:194] Registered new tserver with Master: e66edf8efbec4684bd88334d0d2d3943 (127.10.118.67:42043)
I20260812 03:05:14.242535 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:05:14.242787 4315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.118.67@KRBTEST.COM'} at 127.10.118.67:44177
I20260812 03:05:14.278234 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.258064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.10.118.126:38933)
0812 03:05:14.258873 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.258911 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.259676 (+ 765us) server_negotiation.cc:407] Connection header received
0812 03:05:14.260051 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.260073 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.260261 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.260429 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:14.262760 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.263681 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.265543 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.266129 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.266331 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.269669 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:14.269690 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:14.269708 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:14.269806 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:14.273162 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:14.274121 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:14.274142 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:14.274158 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:14.274341 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:14.274882 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:14.274900 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:14.274915 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:14.275398 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:14.275691 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.276332 (+ 641us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.276748 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260812 03:05:14.285099 4315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:05:14.307973 4315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:05:14.355427 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.334957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.10.118.65:43301)
0812 03:05:14.337095 (+ 2138us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.337118 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.338290 (+ 1172us) server_negotiation.cc:407] Connection header received
0812 03:05:14.338776 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.338798 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.339180 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.339440 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:14.341201 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.343409 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.348346 (+ 4937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.350992 (+ 2646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.351393 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.351985 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.352729 (+ 744us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.353165 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1691,"thread_start_us":2111,"threads_started":1}
I20260812 03:05:14.357730 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.334577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43301 (local address 127.0.0.1:57532)
0812 03:05:14.338041 (+ 3464us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.338100 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:14.338477 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:14.340033 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:14.340066 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:14.340183 (+ 117us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:14.340835 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.340892 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.343822 (+ 2930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.343866 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.347885 (+ 4019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.347960 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.351435 (+ 3475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.351482 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.352065 (+ 583us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.352295 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 03:05:14.352590 (+ 295us) client_negotiation.cc:240] Negotiation successful
0812 03:05:14.355963 (+ 3373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3045,"thread_start_us":2654,"threads_started":1}
I20260812 03:05:14.362629 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 3abccb64b49c40edb71b5315067e048e (DEFAULT_TABLE table=test-table [id=5ba31d17957540e59a89b20869bdc9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:14.364646 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abccb64b49c40edb71b5315067e048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:14.371411 5236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.344263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:39591 (local address 127.0.0.1:57586)
0812 03:05:14.349713 (+ 5450us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.349782 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:14.350069 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:14.357822 (+ 7753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:14.357859 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:14.358468 (+ 609us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:14.359312 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.359359 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.362428 (+ 3069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.362463 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.365493 (+ 3030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.365551 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.368257 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.368287 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.368754 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.368903 (+ 149us) client_negotiation.cc:769] Sending connection context
0812 03:05:14.369033 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 03:05:14.369353 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4922,"mutex_wait_us":31,"spinlock_wait_cycles":155008,"thread_start_us":4606,"threads_started":1}
I20260812 03:05:14.372277 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.338405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42043 (local address 127.0.0.1:56298)
0812 03:05:14.340536 (+ 2131us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.340875 (+ 339us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:14.341124 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:14.357842 (+ 16718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:14.357862 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:14.357986 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:14.358706 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.358765 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.361131 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.361162 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.363905 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.363944 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.365830 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.365862 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.366273 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.366403 (+ 130us) client_negotiation.cc:769] Sending connection context
0812 03:05:14.366556 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 03:05:14.367862 (+ 1306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"mutex_wait_us":177,"thread_start_us":1342,"threads_started":1}
I20260812 03:05:14.375558 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.340939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.10.118.66:39591)
0812 03:05:14.356603 (+ 15664us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.356644 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.356700 (+ 56us) server_negotiation.cc:407] Connection header received
0812 03:05:14.356922 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.356955 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.357319 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.359578 (+ 2259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:14.360141 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.362150 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.365873 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.367857 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.373007 (+ 5150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.373472 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.373615 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.373921 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15161,"thread_start_us":16380,"threads_started":1}
I20260812 03:05:14.376196 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.338808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.10.118.67:42043)
0812 03:05:14.356688 (+ 17880us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.356711 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.356751 (+ 40us) server_negotiation.cc:407] Connection header received
0812 03:05:14.356964 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.356995 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.357358 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.358661 (+ 1303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:14.359280 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.360830 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.364184 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.365479 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.365796 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.366240 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.366649 (+ 409us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.366937 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17485,"thread_start_us":18520,"threads_started":1}
I20260812 03:05:14.380220 4813 tablet_service.cc:1467] Processing CreateTablet for tablet 3abccb64b49c40edb71b5315067e048e (DEFAULT_TABLE table=test-table [id=5ba31d17957540e59a89b20869bdc9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:14.380713 5079 tablet_service.cc:1467] Processing CreateTablet for tablet 3abccb64b49c40edb71b5315067e048e (DEFAULT_TABLE table=test-table [id=5ba31d17957540e59a89b20869bdc9df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:14.382670 5079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abccb64b49c40edb71b5315067e048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:14.384685 4813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3abccb64b49c40edb71b5315067e048e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:14.398386 5238 tablet_bootstrap.cc:492] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48: Bootstrap starting.
I20260812 03:05:14.406071 5238 tablet_bootstrap.cc:654] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:14.408445 5238 log.cc:824] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:14.409384 5239 tablet_bootstrap.cc:492] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Bootstrap starting.
I20260812 03:05:14.414999 5238 tablet_bootstrap.cc:492] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48: No bootstrap required, opened a new log
I20260812 03:05:14.415581 5238 ts_tablet_manager.cc:1397] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20260812 03:05:14.416265 5243 tablet_bootstrap.cc:492] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Bootstrap starting.
I20260812 03:05:14.416893 5239 tablet_bootstrap.cc:654] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:14.418586 5239 log.cc:824] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:14.423121 5243 tablet_bootstrap.cc:654] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:14.424192 5239 tablet_bootstrap.cc:492] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: No bootstrap required, opened a new log
I20260812 03:05:14.424667 5239 ts_tablet_manager.cc:1397] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260812 03:05:14.426613 5243 log.cc:824] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:14.432871 5243 tablet_bootstrap.cc:492] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: No bootstrap required, opened a new log
I20260812 03:05:14.433336 5243 ts_tablet_manager.cc:1397] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20260812 03:05:14.446429 5238 raft_consensus.cc:348] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.446869 5239 raft_consensus.cc:348] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.447533 5238 raft_consensus.cc:374] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:14.447853 5239 raft_consensus.cc:374] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:14.448050 5238 raft_consensus.cc:729] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c8358c5491423b8021429267bd9a48, State: Initialized, Role: FOLLOWER
I20260812 03:05:14.448264 5239 raft_consensus.cc:729] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66edf8efbec4684bd88334d0d2d3943, State: Initialized, Role: FOLLOWER
I20260812 03:05:14.449112 5239 consensus_queue.cc:260] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.449124 5238 consensus_queue.cc:260] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.453378 5238 ts_tablet_manager.cc:1428] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48: Time spent starting tablet: real 0.037s user 0.032s sys 0.004s
I20260812 03:05:14.457242 5201 heartbeater.cc:503] Master 127.10.118.126:38933 was elected leader, sending a full tablet report...
I20260812 03:05:14.460834 5239 ts_tablet_manager.cc:1428] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Time spent starting tablet: real 0.036s user 0.024s sys 0.009s
I20260812 03:05:14.462558 5243 raft_consensus.cc:348] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.463526 5243 raft_consensus.cc:374] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:14.463832 5243 raft_consensus.cc:729] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e3b3fc78784a15a900dde5612efcf4, State: Initialized, Role: FOLLOWER
I20260812 03:05:14.464648 5243 consensus_queue.cc:260] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.468837 5243 ts_tablet_manager.cc:1428] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Time spent starting tablet: real 0.035s user 0.034s sys 0.002s
W20260812 03:05:14.540634 4936 tablet.cc:2369] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:05:14.628273 5250 raft_consensus.cc:482] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:05:14.628822 5250 raft_consensus.cc:504] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.631428 5250 leader_election.cc:288] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67c8358c5491423b8021429267bd9a48 (127.10.118.65:43301), e66edf8efbec4684bd88334d0d2d3943 (127.10.118.67:42043)
W20260812 03:05:14.637174 4682 tablet.cc:2369] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:05:14.645519 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.632102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:36009 (local address 127.10.118.65:43301)
0812 03:05:14.632596 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.632615 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.634812 (+ 2197us) server_negotiation.cc:407] Connection header received
0812 03:05:14.635187 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.635215 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.635445 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.635659 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:14.637208 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.638572 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.641642 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.642694 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.643023 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.643196 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.643903 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.644171 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260812 03:05:14.645663 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.631945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43301 (local address 127.10.118.66:36009)
0812 03:05:14.634566 (+ 2621us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.634620 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:14.634898 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:14.635915 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:14.635954 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:14.636120 (+ 166us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:14.636932 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.636974 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.638854 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.638886 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.641360 (+ 2474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.641409 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.643061 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.643093 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.643561 (+ 468us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.643673 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 03:05:14.643782 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 03:05:14.643999 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2141,"mutex_wait_us":33,"thread_start_us":1835,"threads_started":1}
I20260812 03:05:14.647449 4589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abccb64b49c40edb71b5315067e048e" candidate_uuid: "05e3b3fc78784a15a900dde5612efcf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c8358c5491423b8021429267bd9a48" is_pre_election: true
I20260812 03:05:14.648396 4589 raft_consensus.cc:2454] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e3b3fc78784a15a900dde5612efcf4 in term 0.
I20260812 03:05:14.649091 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.635053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:50537 (local address 127.10.118.67:42043)
0812 03:05:14.635676 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.635722 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.637362 (+ 1640us) server_negotiation.cc:407] Connection header received
0812 03:05:14.637918 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.637947 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.638234 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.638387 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:14.639901 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.640816 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.643618 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.645751 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.646107 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.646347 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.647454 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.647738 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260812 03:05:14.649091 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.634802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:42043 (local address 127.10.118.66:50537)
0812 03:05:14.637188 (+ 2386us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.637250 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:14.637599 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:14.638450 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:14.638469 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:14.638541 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:14.639630 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.639678 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.641061 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.641092 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.643296 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:14.643345 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.646592 (+ 3247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.646622 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:14.647047 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.647199 (+ 152us) client_negotiation.cc:769] Sending connection context
0812 03:05:14.647357 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 03:05:14.647630 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1890,"mutex_wait_us":205,"thread_start_us":1549,"threads_started":1}
I20260812 03:05:14.649894 4727 leader_election.cc:302] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e3b3fc78784a15a900dde5612efcf4, 67c8358c5491423b8021429267bd9a48; no voters:
I20260812 03:05:14.650607 5250 raft_consensus.cc:2781] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:05:14.650903 5250 raft_consensus.cc:482] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:05:14.651163 5250 raft_consensus.cc:3037] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:14.651854 5114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abccb64b49c40edb71b5315067e048e" candidate_uuid: "05e3b3fc78784a15a900dde5612efcf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e66edf8efbec4684bd88334d0d2d3943" is_pre_election: true
I20260812 03:05:14.652724 5114 raft_consensus.cc:2454] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e3b3fc78784a15a900dde5612efcf4 in term 0.
I20260812 03:05:14.655973 5250 raft_consensus.cc:504] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.657326 5250 leader_election.cc:288] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [CANDIDATE]: Term 1 election: Requested vote from peers 67c8358c5491423b8021429267bd9a48 (127.10.118.65:43301), e66edf8efbec4684bd88334d0d2d3943 (127.10.118.67:42043)
I20260812 03:05:14.658164 4589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abccb64b49c40edb71b5315067e048e" candidate_uuid: "05e3b3fc78784a15a900dde5612efcf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67c8358c5491423b8021429267bd9a48"
I20260812 03:05:14.658375 5114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3abccb64b49c40edb71b5315067e048e" candidate_uuid: "05e3b3fc78784a15a900dde5612efcf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e66edf8efbec4684bd88334d0d2d3943"
I20260812 03:05:14.658708 4589 raft_consensus.cc:3037] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:14.658890 5114 raft_consensus.cc:3037] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:14.665850 5114 raft_consensus.cc:2454] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e3b3fc78784a15a900dde5612efcf4 in term 1.
I20260812 03:05:14.665850 4589 raft_consensus.cc:2454] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e3b3fc78784a15a900dde5612efcf4 in term 1.
I20260812 03:05:14.666810 4727 leader_election.cc:302] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e3b3fc78784a15a900dde5612efcf4, e66edf8efbec4684bd88334d0d2d3943; no voters:
I20260812 03:05:14.667400 5250 raft_consensus.cc:2781] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:14.669000 5250 raft_consensus.cc:686] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 1 LEADER]: Becoming Leader. State: Replica: 05e3b3fc78784a15a900dde5612efcf4, State: Running, Role: LEADER
I20260812 03:05:14.669765 5250 consensus_queue.cc:237] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } }
I20260812 03:05:14.682152 4314 catalog_manager.cc:5495] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e3b3fc78784a15a900dde5612efcf4 (127.10.118.66). New cstate: current_term: 1 leader_uuid: "05e3b3fc78784a15a900dde5612efcf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e3b3fc78784a15a900dde5612efcf4" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 39591 } health_report { overall_health: HEALTHY } } }
W20260812 03:05:14.700783 5204 tablet.cc:2369] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:05:14.739498 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.730566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.10.118.66:39591)
0812 03:05:14.731080 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:14.731100 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:14.731137 (+ 37us) server_negotiation.cc:407] Connection header received
0812 03:05:14.731476 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:14.731496 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:14.731747 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:14.731894 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:05:14.732628 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.733628 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.735381 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:14.736156 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:14.736351 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:14.737309 (+ 958us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:05:14.737821 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:05:14.737971 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:14.738214 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 03:05:14.738355 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260812 03:05:14.751709 5114 raft_consensus.cc:1261] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 1 FOLLOWER]: Refusing update from remote peer 05e3b3fc78784a15a900dde5612efcf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:05:14.752231 4589 raft_consensus.cc:1261] T 3abccb64b49c40edb71b5315067e048e P 67c8358c5491423b8021429267bd9a48 [term 1 FOLLOWER]: Refusing update from remote peer 05e3b3fc78784a15a900dde5612efcf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:05:14.753501 5255 consensus_queue.cc:1035] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e66edf8efbec4684bd88334d0d2d3943" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 42043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:05:14.754139 5250 consensus_queue.cc:1035] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67c8358c5491423b8021429267bd9a48" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:05:14.774492 5260 mvcc.cc:205] Tried to move back new op lower bound from 7317520034808467456 to 7317520034494566400. Current Snapshot: MvccSnapshot[applied={T|T < 7317520034808467456}]
I20260812 03:05:14.789024 5269 mvcc.cc:205] Tried to move back new op lower bound from 7317520034808467456 to 7317520034494566400. Current Snapshot: MvccSnapshot[applied={T|T < 7317520034808467456 or (T in {7317520034808467456})}]
I20260812 03:05:14.800976 4313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:05:14.801589 4313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:05:14.810670 4313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bf7cfe8968d48e996e796a8f60114a3: Sending DeleteTablet for 3 replicas of tablet 3abccb64b49c40edb71b5315067e048e
I20260812 03:05:14.812593 4554 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abccb64b49c40edb71b5315067e048e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:14 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57532
I20260812 03:05:14.813180 5079 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abccb64b49c40edb71b5315067e048e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:14 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:56298
I20260812 03:05:14.815248 4813 tablet_service.cc:1514] Processing DeleteTablet for tablet 3abccb64b49c40edb71b5315067e048e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:14 UTC) from {username='kudu', principal='kudu/127.10.118.126@KRBTEST.COM'} at 127.0.0.1:57586
I20260812 03:05:14.816608 5288 tablet_replica.cc:330] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: stopping tablet replica
I20260812 03:05:14.816836 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 4443
I20260812 03:05:14.819449 5288 raft_consensus.cc:2229] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:05:14.820297 5288 raft_consensus.cc:2258] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:05:14.829208 5289 tablet_replica.cc:330] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: stopping tablet replica
I20260812 03:05:14.831517 5288 ts_tablet_manager.cc:1905] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:05:14.829981 5289 raft_consensus.cc:2229] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:05:14.837117 5289 raft_consensus.cc:2258] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:05:14.840859 5289 ts_tablet_manager.cc:1905] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:05:14.849632 5288 ts_tablet_manager.cc:1918] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:05:14.850081 5288 log.cc:1196] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/3abccb64b49c40edb71b5315067e048e
I20260812 03:05:14.850934 5288 ts_tablet_manager.cc:1939] T 3abccb64b49c40edb71b5315067e048e P e66edf8efbec4684bd88334d0d2d3943: Deleting consensus metadata
I20260812 03:05:14.854281 4288 catalog_manager.cc:4841] TS e66edf8efbec4684bd88334d0d2d3943 (127.10.118.67:42043): tablet 3abccb64b49c40edb71b5315067e048e (table test-table [id=5ba31d17957540e59a89b20869bdc9df]) successfully deleted
I20260812 03:05:14.858012 5289 ts_tablet_manager.cc:1918] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:05:14.858508 5289 log.cc:1196] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/3abccb64b49c40edb71b5315067e048e
I20260812 03:05:14.859390 5289 ts_tablet_manager.cc:1939] T 3abccb64b49c40edb71b5315067e048e P 05e3b3fc78784a15a900dde5612efcf4: Deleting consensus metadata
I20260812 03:05:14.862615 4288 catalog_manager.cc:4841] TS 05e3b3fc78784a15a900dde5612efcf4 (127.10.118.66:39591): tablet 3abccb64b49c40edb71b5315067e048e (table test-table [id=5ba31d17957540e59a89b20869bdc9df]) successfully deleted
W20260812 03:05:14.872310 4288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:43301 (error 108)
I20260812 03:05:14.872478 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 4694
I20260812 03:05:14.885344 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.883650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43301 (local address 127.0.0.1:57542)
0812 03:05:14.884657 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.884835 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":373,"threads_started":1}
W20260812 03:05:14.885748 4288 catalog_manager.cc:4584] TS 67c8358c5491423b8021429267bd9a48 (127.10.118.65:43301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3abccb64b49c40edb71b5315067e048e: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43301: connect: Connection refused (error 111)
I20260812 03:05:14.907589 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 4946
I20260812 03:05:14.941033 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.939950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43301 (local address 127.0.0.1:57546)
0812 03:05:14.940451 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.940615 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 03:05:14.943382 5291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:14.942418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:43301 (local address 127.0.0.1:57562)
0812 03:05:14.942849 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:14.942994 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 03:05:14.945087 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 4257
2026-08-12T03:05:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (10394 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T03:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:15Z Disabled control of system clock
I20260812 03:05:15.047158 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:34727
--webserver_interface=127.10.118.126
--webserver_port=0
--builtin_ntp_servers=127.10.118.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 03:05:15.356647 5298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:15.357278 5298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:15.357822 5298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:15.388736 5298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:15.389133 5298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:15.389369 5298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:15.389632 5298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:15.389882 5298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:15.424794 5298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:34727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:15.426147 5298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:15.427784 5298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:15.444990 5316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:15.445003 5310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:15.446974 5298 server_base.cc:1034] running on GCE node
W20260812 03:05:15.445634 5308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:16.658227 5298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:16.660974 5298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:16.662386 5298 hybrid_clock.cc:648] HybridClock initialized: now 1786503916662345 us; error 54 us; skew 500 ppm
I20260812 03:05:16.663246 5298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:05:16.674032 5298 webserver.cc:460] Webserver started at http://127.10.118.126:42649/ using document root <none> and password file <none>
I20260812 03:05:16.674964 5298 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:16.675148 5298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:16.675581 5298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:16.680969 5298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "ed364f4f72674976ba3e6146626778f5"
format_stamp: "Formatted at 2026-08-12 03:05:16 on dist-test-slave-zpvx"
server_key: "6ea1d7be6aef85b82cb0809ddf5ed6c9"
server_key_iv: "ba65012c884093394b78d1a5eaf35761"
server_key_version: "encryptionkey@0"
I20260812 03:05:16.682075 5298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "ed364f4f72674976ba3e6146626778f5"
format_stamp: "Formatted at 2026-08-12 03:05:16 on dist-test-slave-zpvx"
server_key: "6ea1d7be6aef85b82cb0809ddf5ed6c9"
server_key_iv: "ba65012c884093394b78d1a5eaf35761"
server_key_version: "encryptionkey@0"
I20260812 03:05:16.691622 5298 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260812 03:05:16.699095 5329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:16.700074 5298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 03:05:16.700490 5298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "ed364f4f72674976ba3e6146626778f5"
format_stamp: "Formatted at 2026-08-12 03:05:16 on dist-test-slave-zpvx"
server_key: "6ea1d7be6aef85b82cb0809ddf5ed6c9"
server_key_iv: "ba65012c884093394b78d1a5eaf35761"
server_key_version: "encryptionkey@0"
I20260812 03:05:16.700863 5298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:16.747565 5298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:16.749168 5298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:16.749651 5298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:16.818918 5298 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:34727
I20260812 03:05:16.819026 5417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:34727 every 8 connection(s)
I20260812 03:05:16.821755 5298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:16.826963 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:16.831161 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 5298
I20260812 03:05:16.831714 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:16.833184 10713 external_mini_cluster.cc:1428] Setting key 448bfd9440c5af92069aaab7f574fce3
I20260812 03:05:16.853042 5419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5: Bootstrap starting.
I20260812 03:05:16.860080 5419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:16.862408 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:16.835937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43740 (local address 127.10.118.126:34727)
0812 03:05:16.838846 (+ 2909us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:16.838870 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:16.838924 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:05:16.844340 (+ 5416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:16.844424 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:16.845003 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:16.845501 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:16.847842 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:16.850512 (+ 2670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:16.853959 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:16.855314 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:16.855705 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:16.857506 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:16.857544 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:16.857588 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:16.858691 (+ 1103us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:16.858821 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:16.858884 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:16.859056 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:16.859357 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 03:05:16.859731 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2389,"thread_start_us":1901,"threads_started":1}
I20260812 03:05:16.862486 5419 log.cc:824] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:16.867653 5419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5: No bootstrap required, opened a new log
I20260812 03:05:16.886996 5419 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed364f4f72674976ba3e6146626778f5" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34727 } }
I20260812 03:05:16.887655 5419 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:16.887863 5419 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed364f4f72674976ba3e6146626778f5, State: Initialized, Role: FOLLOWER
I20260812 03:05:16.888533 5419 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed364f4f72674976ba3e6146626778f5" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34727 } }
I20260812 03:05:16.889029 5419 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:16.889283 5419 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:16.889533 5419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:16.893497 5419 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed364f4f72674976ba3e6146626778f5" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34727 } }
I20260812 03:05:16.894246 5419 leader_election.cc:302] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed364f4f72674976ba3e6146626778f5; no voters:
I20260812 03:05:16.895885 5419 leader_election.cc:288] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:16.896286 5429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:16.898896 5429 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [term 1 LEADER]: Becoming Leader. State: Replica: ed364f4f72674976ba3e6146626778f5, State: Running, Role: LEADER
I20260812 03:05:16.899611 5429 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed364f4f72674976ba3e6146626778f5" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34727 } }
I20260812 03:05:16.900593 5419 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:16.912770 5432 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed364f4f72674976ba3e6146626778f5. Latest consensus state: current_term: 1 leader_uuid: "ed364f4f72674976ba3e6146626778f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed364f4f72674976ba3e6146626778f5" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34727 } } }
I20260812 03:05:16.913600 5432 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:16.913374 5430 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed364f4f72674976ba3e6146626778f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed364f4f72674976ba3e6146626778f5" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 34727 } } }
I20260812 03:05:16.913954 5430 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:16.917824 5449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:16.928579 5449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:16.944999 5449 catalog_manager.cc:1347] Generated new cluster ID: 1df9122053be4aa79cc847ef802d9e44
I20260812 03:05:16.945333 5449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:16.970521 5449 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:16.972297 5449 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:16.983453 5449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5: Generated new TSK 0
I20260812 03:05:16.984350 5449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:16.995272 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.65:0
--local_ip_for_outbound_sockets=127.10.118.65
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34727
--builtin_ntp_servers=127.10.118.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 03:05:17.316893 5462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:17.317466 5462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:17.318034 5462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:17.350973 5462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:17.351270 5462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:17.352278 5462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.65
I20260812 03:05:17.388947 5462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.118.65
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.65
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:17.390174 5462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:17.391834 5462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:17.412499 5481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:17.412546 5477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:17.414175 5485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:18.592945 5482 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:05:18.593130 5462 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:05:18.597038 5462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:18.599800 5462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:18.601281 5462 hybrid_clock.cc:648] HybridClock initialized: now 1786503918601243 us; error 51 us; skew 500 ppm
I20260812 03:05:18.602176 5462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:05:18.609688 5462 webserver.cc:460] Webserver started at http://127.10.118.65:44999/ using document root <none> and password file <none>
I20260812 03:05:18.610705 5462 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:18.610932 5462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:18.611423 5462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:18.617240 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data/instance:
uuid: "fcd494c8686d446b80b456554024d021"
format_stamp: "Formatted at 2026-08-12 03:05:18 on dist-test-slave-zpvx"
server_key: "cc8f17a8b00cd6605515ec7519f9c22d"
server_key_iv: "51a83092c9abf1bc12ad2bc62277479b"
server_key_version: "encryptionkey@0"
I20260812 03:05:18.618502 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance:
uuid: "fcd494c8686d446b80b456554024d021"
format_stamp: "Formatted at 2026-08-12 03:05:18 on dist-test-slave-zpvx"
server_key: "cc8f17a8b00cd6605515ec7519f9c22d"
server_key_iv: "51a83092c9abf1bc12ad2bc62277479b"
server_key_version: "encryptionkey@0"
I20260812 03:05:18.626328 5462 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20260812 03:05:18.632078 5498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:18.633249 5462 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 03:05:18.633610 5462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/wal
uuid: "fcd494c8686d446b80b456554024d021"
format_stamp: "Formatted at 2026-08-12 03:05:18 on dist-test-slave-zpvx"
server_key: "cc8f17a8b00cd6605515ec7519f9c22d"
server_key_iv: "51a83092c9abf1bc12ad2bc62277479b"
server_key_version: "encryptionkey@0"
I20260812 03:05:18.633996 5462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:18.673873 5462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:18.675374 5462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:18.675839 5462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:18.678485 5462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:18.682533 5462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:18.682734 5462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:18.683058 5462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:18.683200 5462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:18.825317 5462 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.65:37633
I20260812 03:05:18.825438 5715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.65:37633 every 8 connection(s)
I20260812 03:05:18.828029 5462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/data/info.pb
I20260812 03:05:18.838119 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 5462
I20260812 03:05:18.838564 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-0/wal/instance
I20260812 03:05:18.839581 10713 external_mini_cluster.cc:1428] Setting key e6a53d829a26fc4a7f3fc65f33d3e807
I20260812 03:05:18.845113 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.66:0
--local_ip_for_outbound_sockets=127.10.118.66
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34727
--builtin_ntp_servers=127.10.118.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 03:05:18.851898 5721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:18.832323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.65:57975 (local address 127.10.118.126:34727)
0812 03:05:18.833873 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:18.833911 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:18.835612 (+ 1701us) server_negotiation.cc:407] Connection header received
0812 03:05:18.840629 (+ 5017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:18.840661 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:18.840847 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:18.841031 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:18.843088 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:18.844006 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:18.846295 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:18.846894 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:18.847188 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:18.849122 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:18.849143 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:18.849160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:18.849205 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:18.849270 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:18.849305 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:18.849455 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:18.849804 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 03:05:18.850018 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":739,"threads_started":1}
I20260812 03:05:18.852793 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:18.832777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34727 (local address 127.10.118.65:57975)
0812 03:05:18.835382 (+ 2605us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:18.835451 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:18.840238 (+ 4787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:18.841318 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:18.841352 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:18.841548 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:18.842790 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:18.842855 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:18.844273 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:18.844304 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:18.846036 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:18.846083 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:18.847238 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:18.847281 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:18.847700 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:18.848598 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:05:18.848614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:18.848683 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:05:18.848795 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:18.849478 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:18.849529 (+ 51us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:18.849551 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 03:05:18.849735 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 03:05:18.850195 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"thread_start_us":1270,"threads_started":1}
I20260812 03:05:18.856228 5719 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34727
I20260812 03:05:18.856873 5719 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:18.858268 5719 heartbeater.cc:511] Master 127.10.118.126:34727 requested a full tablet report, sending...
I20260812 03:05:18.861617 5362 ts_manager.cc:194] Registered new tserver with Master: fcd494c8686d446b80b456554024d021 (127.10.118.65:37633)
I20260812 03:05:18.864914 5362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.65:57975
W20260812 03:05:19.152163 5724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:19.152742 5724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:19.153329 5724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:19.184146 5724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:19.184430 5724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:19.185304 5724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.66
I20260812 03:05:19.221380 5724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.118.66
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.66
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:19.222657 5724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:19.224248 5724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:19.244268 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:19.870388 5719 heartbeater.cc:503] Master 127.10.118.126:34727 was elected leader, sending a full tablet report...
W20260812 03:05:19.244285 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:19.245995 5724 server_base.cc:1034] running on GCE node
W20260812 03:05:19.244308 5739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:20.442739 5724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:20.445099 5724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:20.446538 5724 hybrid_clock.cc:648] HybridClock initialized: now 1786503920446510 us; error 53 us; skew 500 ppm
I20260812 03:05:20.447454 5724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:05:20.454713 5724 webserver.cc:460] Webserver started at http://127.10.118.66:35397/ using document root <none> and password file <none>
I20260812 03:05:20.455694 5724 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:20.455940 5724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:20.456482 5724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:20.462318 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data/instance:
uuid: "a03aec67e0484ff88f41cf7d400dac57"
format_stamp: "Formatted at 2026-08-12 03:05:20 on dist-test-slave-zpvx"
server_key: "a86a22262515d73c5fc48f79ac71f7b3"
server_key_iv: "b21b97867492ef67dbdbb0b0f9ed328e"
server_key_version: "encryptionkey@0"
I20260812 03:05:20.463635 5724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance:
uuid: "a03aec67e0484ff88f41cf7d400dac57"
format_stamp: "Formatted at 2026-08-12 03:05:20 on dist-test-slave-zpvx"
server_key: "a86a22262515d73c5fc48f79ac71f7b3"
server_key_iv: "b21b97867492ef67dbdbb0b0f9ed328e"
server_key_version: "encryptionkey@0"
I20260812 03:05:20.470878 5724 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s
I20260812 03:05:20.476789 5748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:20.477864 5724 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:05:20.478205 5724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal
uuid: "a03aec67e0484ff88f41cf7d400dac57"
format_stamp: "Formatted at 2026-08-12 03:05:20 on dist-test-slave-zpvx"
server_key: "a86a22262515d73c5fc48f79ac71f7b3"
server_key_iv: "b21b97867492ef67dbdbb0b0f9ed328e"
server_key_version: "encryptionkey@0"
I20260812 03:05:20.478612 5724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:20.515477 5724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:20.517238 5724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:20.517706 5724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:20.520409 5724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:20.524657 5724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:20.524889 5724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:20.525208 5724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:20.525403 5724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:20.659009 5724 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.66:42959
I20260812 03:05:20.659109 5954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.66:42959 every 8 connection(s)
I20260812 03:05:20.661909 5724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/data/info.pb
I20260812 03:05:20.663816 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 5724
I20260812 03:05:20.664462 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal/instance
I20260812 03:05:20.666136 10713 external_mini_cluster.cc:1428] Setting key 8240080c0f3ffd1675eea553865bdd99
I20260812 03:05:20.672547 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.118.67:0
--local_ip_for_outbound_sockets=127.10.118.67
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34727
--builtin_ntp_servers=127.10.118.84:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 03:05:20.686125 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:20.666896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:45129 (local address 127.10.118.126:34727)
0812 03:05:20.668749 (+ 1853us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:20.668790 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:20.670320 (+ 1530us) server_negotiation.cc:407] Connection header received
0812 03:05:20.675264 (+ 4944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:20.675295 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:20.675490 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:20.675673 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:20.677391 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:20.678317 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:20.680411 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:20.681045 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:20.681339 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:20.683320 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:20.683344 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:20.683360 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:20.683404 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:20.683475 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:20.683509 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:20.683653 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:20.683986 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 03:05:20.684203 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1313,"thread_start_us":994,"threads_started":1}
I20260812 03:05:20.686905 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:20.667307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34727 (local address 127.10.118.66:45129)
0812 03:05:20.670069 (+ 2762us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:20.670136 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:20.674924 (+ 4788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:20.675951 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:20.675987 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:20.676133 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:20.677105 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:20.677151 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:20.678578 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:20.678598 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:20.680133 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:20.680171 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:20.681382 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:20.681416 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:20.681850 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:20.682797 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:05:20.682813 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:20.682873 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:05:20.682994 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:20.683679 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:20.683726 (+ 47us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:20.683746 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 03:05:20.683926 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 03:05:20.684397 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2202,"thread_start_us":1505,"threads_started":1}
I20260812 03:05:20.689815 5955 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34727
I20260812 03:05:20.690363 5955 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:20.691623 5955 heartbeater.cc:511] Master 127.10.118.126:34727 requested a full tablet report, sending...
I20260812 03:05:20.694911 5362 ts_manager.cc:194] Registered new tserver with Master: a03aec67e0484ff88f41cf7d400dac57 (127.10.118.66:42959)
I20260812 03:05:20.696602 5362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.66:45129
W20260812 03:05:20.990396 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:20.990943 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:20.991518 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:21.023339 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:21.023615 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:21.024501 5961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.118.67
I20260812 03:05:21.059068 5961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.118.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.118.67
--webserver_port=0
--tserver_master_addrs=127.10.118.126:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.118.67
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:21.060276 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:21.061964 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:21.080390 5976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:21.700438 5955 heartbeater.cc:503] Master 127.10.118.126:34727 was elected leader, sending a full tablet report...
W20260812 03:05:21.080426 5975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:21.080425 5979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:22.281657 5978 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1199 milliseconds
I20260812 03:05:22.281774 5961 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:05:22.283075 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:22.285873 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:22.287307 5961 hybrid_clock.cc:648] HybridClock initialized: now 1786503922287266 us; error 59 us; skew 500 ppm
I20260812 03:05:22.288069 5961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 03:05:22.298275 5961 webserver.cc:460] Webserver started at http://127.10.118.67:42369/ using document root <none> and password file <none>
I20260812 03:05:22.299291 5961 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:22.299475 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:22.299962 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:22.305323 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data/instance:
uuid: "e73e6d9dd8514001b82505a7043afd41"
format_stamp: "Formatted at 2026-08-12 03:05:22 on dist-test-slave-zpvx"
server_key: "31d51fc94715df739cdbccd768ad84ba"
server_key_iv: "640a5786d54a288808c7d0ba31797056"
server_key_version: "encryptionkey@0"
I20260812 03:05:22.306411 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance:
uuid: "e73e6d9dd8514001b82505a7043afd41"
format_stamp: "Formatted at 2026-08-12 03:05:22 on dist-test-slave-zpvx"
server_key: "31d51fc94715df739cdbccd768ad84ba"
server_key_iv: "640a5786d54a288808c7d0ba31797056"
server_key_version: "encryptionkey@0"
I20260812 03:05:22.313546 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:05:22.319146 5987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:22.320237 5961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 03:05:22.320590 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal
uuid: "e73e6d9dd8514001b82505a7043afd41"
format_stamp: "Formatted at 2026-08-12 03:05:22 on dist-test-slave-zpvx"
server_key: "31d51fc94715df739cdbccd768ad84ba"
server_key_iv: "640a5786d54a288808c7d0ba31797056"
server_key_version: "encryptionkey@0"
I20260812 03:05:22.321009 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:22.361392 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:22.362998 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:22.363473 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:22.365989 5961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 03:05:22.370013 5961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 03:05:22.370251 5961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:22.370524 5961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 03:05:22.370707 5961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:22.507066 5961 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.67:41863
I20260812 03:05:22.507150 6192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.67:41863 every 8 connection(s)
I20260812 03:05:22.509718 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/data/info.pb
I20260812 03:05:22.516263 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 5961
I20260812 03:05:22.516887 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal/instance
I20260812 03:05:22.518409 10713 external_mini_cluster.cc:1428] Setting key 1bff35e36d3ff559b6f1e6fd4287ae90
I20260812 03:05:22.533684 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.514029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.67:60847 (local address 127.10.118.126:34727)
0812 03:05:22.515271 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:22.515330 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:22.517776 (+ 2446us) server_negotiation.cc:407] Connection header received
0812 03:05:22.524081 (+ 6305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:22.524100 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:22.524270 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:22.524470 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:22.525935 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.526908 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.529042 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.529679 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.529873 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.531680 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:22.531700 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:22.531716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:22.531759 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:22.531822 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:22.531855 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:22.531964 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:22.532298 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 03:05:22.532503 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":557,"threads_started":1}
I20260812 03:05:22.534409 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.514274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:34727 (local address 127.10.118.67:60847)
0812 03:05:22.517495 (+ 3221us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:22.517605 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:22.523798 (+ 6193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:22.524672 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:22.524695 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:22.524808 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:05:22.525696 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.525739 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.527116 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.527135 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.528826 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.528863 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.529937 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.529961 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.530297 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.531143 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 03:05:22.531161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:05:22.531220 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 03:05:22.531329 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:05:22.532007 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:05:22.532040 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:05:22.532075 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 03:05:22.532238 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 03:05:22.532601 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2467,"thread_start_us":1528,"threads_started":1}
I20260812 03:05:22.536160 6193 heartbeater.cc:347] Connected to a master server at 127.10.118.126:34727
I20260812 03:05:22.536628 6193 heartbeater.cc:464] Registering TS with master...
I20260812 03:05:22.537717 6193 heartbeater.cc:511] Master 127.10.118.126:34727 requested a full tablet report, sending...
I20260812 03:05:22.539798 5362 ts_manager.cc:194] Registered new tserver with Master: e73e6d9dd8514001b82505a7043afd41 (127.10.118.67:41863)
I20260812 03:05:22.541322 5362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.118.67:60847
I20260812 03:05:22.551509 10713 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 03:05:22.573021 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.563041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.10.118.126:34727)
0812 03:05:22.563420 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:22.563438 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:22.564721 (+ 1283us) server_negotiation.cc:407] Connection header received
0812 03:05:22.565022 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:22.565039 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:22.565206 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:22.565334 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:22.566225 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.567140 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.569010 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.569631 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.569834 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.571244 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:22.571267 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:22.571283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:22.571326 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:22.571388 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:22.571422 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:22.571527 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:22.571769 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 03:05:22.571926 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 03:05:22.578610 5362 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 03:05:22.598312 5362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 03:05:22.653225 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.624351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:41863 (local address 127.0.0.1:32824)
0812 03:05:22.634080 (+ 9729us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:22.634186 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:22.634614 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:22.639059 (+ 4445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:22.639088 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:22.639199 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:22.639983 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.640045 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.643781 (+ 3736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.643838 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.646657 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.646696 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.649252 (+ 2556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.649289 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.650962 (+ 1673us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.651138 (+ 176us) client_negotiation.cc:769] Sending connection context
0812 03:05:22.651335 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 03:05:22.651671 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9168,"mutex_wait_us":1199,"thread_start_us":13754,"threads_started":1}
I20260812 03:05:22.659128 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.625921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.66:42959 (local address 127.0.0.1:53372)
0812 03:05:22.642717 (+ 16796us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:22.642777 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:22.643149 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:22.644499 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:22.644536 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:22.644660 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:22.645599 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.645651 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.648050 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.648072 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.652048 (+ 3976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.652084 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.654752 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.654785 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.655395 (+ 610us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.655541 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 03:05:22.655686 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 03:05:22.656025 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12985,"mutex_wait_us":1307,"thread_start_us":7339,"threads_started":1}
I20260812 03:05:22.659680 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.639363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.10.118.65:37633)
0812 03:05:22.642220 (+ 2857us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:22.642243 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:22.642282 (+ 39us) server_negotiation.cc:407] Connection header received
0812 03:05:22.642444 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:22.642470 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:22.642855 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:22.643066 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:22.645215 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.647233 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.652538 (+ 5305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.654454 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.654716 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.655290 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:22.657308 (+ 2018us) server_negotiation.cc:299] Negotiation successful
0812 03:05:22.658015 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2364,"thread_start_us":1983,"threads_started":1}
I20260812 03:05:22.659948 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.627725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.10.118.66:42959)
0812 03:05:22.637776 (+ 10051us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:22.637819 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:22.642921 (+ 5102us) server_negotiation.cc:407] Connection header received
0812 03:05:22.643466 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:22.643514 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:22.643869 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:22.644077 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:22.645949 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.647804 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.652315 (+ 4511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.654412 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.654711 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.655245 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:22.655834 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 03:05:22.656133 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9355,"thread_start_us":8913,"threads_started":1}
I20260812 03:05:22.661613 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.639024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37633 (local address 127.0.0.1:33144)
0812 03:05:22.641391 (+ 2367us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:22.641733 (+ 342us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:22.641983 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:22.643604 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:22.643638 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:22.643755 (+ 117us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:22.644866 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.644917 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.647510 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.647540 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.652132 (+ 4592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.652184 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.655484 (+ 3300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.655505 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.655879 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.656024 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 03:05:22.656160 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 03:05:22.656483 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1922,"mutex_wait_us":1527,"thread_start_us":1660,"threads_started":1}
I20260812 03:05:22.655876 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.624843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.10.118.67:41863)
0812 03:05:22.637914 (+ 13071us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:22.637945 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:22.638011 (+ 66us) server_negotiation.cc:407] Connection header received
0812 03:05:22.638207 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:22.638236 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:22.638665 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:22.640330 (+ 1665us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:22.641182 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.643477 (+ 2295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.646945 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.648871 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.649573 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.650120 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:22.654114 (+ 3994us) server_negotiation.cc:299] Negotiation successful
0812 03:05:22.654321 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12509,"thread_start_us":12836,"threads_started":1}
I20260812 03:05:22.665200 5583 tablet_service.cc:1467] Processing CreateTablet for tablet 51bac57af2434955a4277d6cf1661710 (DEFAULT_TABLE table=test-table [id=fba4f245999c4575870b05047d6aaba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:22.665925 5833 tablet_service.cc:1467] Processing CreateTablet for tablet 51bac57af2434955a4277d6cf1661710 (DEFAULT_TABLE table=test-table [id=fba4f245999c4575870b05047d6aaba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:22.667184 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bac57af2434955a4277d6cf1661710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:22.667909 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bac57af2434955a4277d6cf1661710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:22.668666 6073 tablet_service.cc:1467] Processing CreateTablet for tablet 51bac57af2434955a4277d6cf1661710 (DEFAULT_TABLE table=test-table [id=fba4f245999c4575870b05047d6aaba1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 03:05:22.670584 6073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51bac57af2434955a4277d6cf1661710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:22.698740 6224 tablet_bootstrap.cc:492] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Bootstrap starting.
I20260812 03:05:22.702178 6225 tablet_bootstrap.cc:492] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021: Bootstrap starting.
I20260812 03:05:22.703912 6226 tablet_bootstrap.cc:492] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Bootstrap starting.
I20260812 03:05:22.706137 6224 tablet_bootstrap.cc:654] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:22.709206 6224 log.cc:824] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:22.710717 6225 tablet_bootstrap.cc:654] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:22.711273 6226 tablet_bootstrap.cc:654] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:22.713459 6225 log.cc:824] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:22.713990 6226 log.cc:824] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:22.716964 6224 tablet_bootstrap.cc:492] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: No bootstrap required, opened a new log
I20260812 03:05:22.717451 6224 ts_tablet_manager.cc:1397] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.000s
I20260812 03:05:22.719486 6226 tablet_bootstrap.cc:492] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: No bootstrap required, opened a new log
I20260812 03:05:22.719723 6225 tablet_bootstrap.cc:492] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021: No bootstrap required, opened a new log
I20260812 03:05:22.719933 6226 ts_tablet_manager.cc:1397] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260812 03:05:22.720189 6225 ts_tablet_manager.cc:1397] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.007s
I20260812 03:05:22.740401 6226 raft_consensus.cc:348] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.741284 6226 raft_consensus.cc:374] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:22.741608 6226 raft_consensus.cc:729] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73e6d9dd8514001b82505a7043afd41, State: Initialized, Role: FOLLOWER
I20260812 03:05:22.742435 6226 consensus_queue.cc:260] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.745725 6193 heartbeater.cc:503] Master 127.10.118.126:34727 was elected leader, sending a full tablet report...
I20260812 03:05:22.746544 6226 ts_tablet_manager.cc:1428] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Time spent starting tablet: real 0.026s user 0.021s sys 0.003s
I20260812 03:05:22.748921 6224 raft_consensus.cc:348] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.749976 6224 raft_consensus.cc:374] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:22.750303 6224 raft_consensus.cc:729] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a03aec67e0484ff88f41cf7d400dac57, State: Initialized, Role: FOLLOWER
I20260812 03:05:22.751116 6225 raft_consensus.cc:348] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.751212 6224 consensus_queue.cc:260] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.752012 6225 raft_consensus.cc:374] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:22.752351 6225 raft_consensus.cc:729] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd494c8686d446b80b456554024d021, State: Initialized, Role: FOLLOWER
I20260812 03:05:22.753544 6225 consensus_queue.cc:260] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.755565 6224 ts_tablet_manager.cc:1428] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Time spent starting tablet: real 0.038s user 0.025s sys 0.012s
I20260812 03:05:22.757794 6225 ts_tablet_manager.cc:1428] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021: Time spent starting tablet: real 0.037s user 0.029s sys 0.007s
W20260812 03:05:22.764348 6194 tablet.cc:2369] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 03:05:22.836930 5720 tablet.cc:2369] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:05:22.887877 6231 raft_consensus.cc:482] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 03:05:22.888363 6231 raft_consensus.cc:504] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.890828 6231 leader_election.cc:288] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e73e6d9dd8514001b82505a7043afd41 (127.10.118.67:41863), fcd494c8686d446b80b456554024d021 (127.10.118.65:37633)
I20260812 03:05:22.904942 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.891211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.67:41863 (local address 127.10.118.66:48479)
0812 03:05:22.893874 (+ 2663us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:22.893939 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:22.894237 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:22.895122 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:22.895150 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:22.895273 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:22.895901 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.895937 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.897610 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.897644 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.900613 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.900671 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.902344 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.902378 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.903161 (+ 783us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.903318 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 03:05:22.903442 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 03:05:22.903629 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2097,"mutex_wait_us":279,"thread_start_us":1760,"threads_started":1}
I20260812 03:05:22.905206 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.891390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:48479 (local address 127.10.118.67:41863)
0812 03:05:22.891815 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:22.891833 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:22.894081 (+ 2248us) server_negotiation.cc:407] Connection header received
0812 03:05:22.894626 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:22.894657 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:22.894895 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:22.895059 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:22.896159 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.897304 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.900897 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.902040 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.902295 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.902461 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:22.903543 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0812 03:05:22.903820 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 03:05:22.907418 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.894196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.118.66:59489 (local address 127.10.118.65:37633)
0812 03:05:22.894681 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:22.894706 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:22.896808 (+ 2102us) server_negotiation.cc:407] Connection header received
0812 03:05:22.897095 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:22.897122 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:22.897378 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:22.897545 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 03:05:22.899169 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.900128 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.903348 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.904967 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.905580 (+ 613us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.905841 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:22.906004 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 03:05:22.906203 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 03:05:22.907521 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:22.894011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37633 (local address 127.10.118.66:59489)
0812 03:05:22.896645 (+ 2634us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:22.896683 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 03:05:22.896867 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:05:22.897629 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:05:22.897657 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:05:22.897760 (+ 103us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 03:05:22.898900 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.898953 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.900362 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.900419 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.903054 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:05:22.903111 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:05:22.905319 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:05:22.905357 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:05:22.905708 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:22.905799 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 03:05:22.905957 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 03:05:22.906258 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2132,"mutex_wait_us":390,"thread_start_us":1824,"threads_started":1}
I20260812 03:05:22.907238 6119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bac57af2434955a4277d6cf1661710" candidate_uuid: "a03aec67e0484ff88f41cf7d400dac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73e6d9dd8514001b82505a7043afd41" is_pre_election: true
I20260812 03:05:22.908146 6119 raft_consensus.cc:2454] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03aec67e0484ff88f41cf7d400dac57 in term 0.
I20260812 03:05:22.909467 5753 leader_election.cc:302] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03aec67e0484ff88f41cf7d400dac57, e73e6d9dd8514001b82505a7043afd41; no voters:
I20260812 03:05:22.909493 5624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bac57af2434955a4277d6cf1661710" candidate_uuid: "a03aec67e0484ff88f41cf7d400dac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd494c8686d446b80b456554024d021" is_pre_election: true
I20260812 03:05:22.910252 6231 raft_consensus.cc:2781] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 03:05:22.910260 5624 raft_consensus.cc:2454] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a03aec67e0484ff88f41cf7d400dac57 in term 0.
I20260812 03:05:22.910495 6231 raft_consensus.cc:482] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 03:05:22.910827 6231 raft_consensus.cc:3037] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:22.917833 6231 raft_consensus.cc:504] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
W20260812 03:05:22.919236 5956 tablet.cc:2369] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:05:22.919379 6231 leader_election.cc:288] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [CANDIDATE]: Term 1 election: Requested vote from peers e73e6d9dd8514001b82505a7043afd41 (127.10.118.67:41863), fcd494c8686d446b80b456554024d021 (127.10.118.65:37633)
I20260812 03:05:22.919979 6119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bac57af2434955a4277d6cf1661710" candidate_uuid: "a03aec67e0484ff88f41cf7d400dac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e73e6d9dd8514001b82505a7043afd41"
I20260812 03:05:22.920308 5624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51bac57af2434955a4277d6cf1661710" candidate_uuid: "a03aec67e0484ff88f41cf7d400dac57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcd494c8686d446b80b456554024d021"
I20260812 03:05:22.920500 6119 raft_consensus.cc:3037] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:22.920804 5624 raft_consensus.cc:3037] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:22.925134 6119 raft_consensus.cc:2454] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03aec67e0484ff88f41cf7d400dac57 in term 1.
I20260812 03:05:22.925386 5624 raft_consensus.cc:2454] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a03aec67e0484ff88f41cf7d400dac57 in term 1.
I20260812 03:05:22.925966 5753 leader_election.cc:302] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a03aec67e0484ff88f41cf7d400dac57, e73e6d9dd8514001b82505a7043afd41; no voters:
I20260812 03:05:22.926481 6231 raft_consensus.cc:2781] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:22.927999 6231 raft_consensus.cc:686] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 1 LEADER]: Becoming Leader. State: Replica: a03aec67e0484ff88f41cf7d400dac57, State: Running, Role: LEADER
I20260812 03:05:22.928750 6231 consensus_queue.cc:237] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } }
I20260812 03:05:22.939409 5361 catalog_manager.cc:5495] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 reported cstate change: term changed from 0 to 1, leader changed from <none> to a03aec67e0484ff88f41cf7d400dac57 (127.10.118.66). New cstate: current_term: 1 leader_uuid: "a03aec67e0484ff88f41cf7d400dac57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a03aec67e0484ff88f41cf7d400dac57" member_type: VOTER last_known_addr { host: "127.10.118.66" port: 42959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 } health_report { overall_health: UNKNOWN } } }
I20260812 03:05:23.028841 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:23.019947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.10.118.66:42959)
0812 03:05:23.020486 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:23.020527 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:23.020565 (+ 38us) server_negotiation.cc:407] Connection header received
0812 03:05:23.020915 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:23.020935 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:23.021165 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:23.021374 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:05:23.022195 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:23.023051 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:23.024845 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:23.025429 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:23.025703 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:23.026551 (+ 848us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:05:23.027040 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:05:23.027248 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:23.027505 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 03:05:23.027682 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 03:05:23.041615 5624 raft_consensus.cc:1261] T 51bac57af2434955a4277d6cf1661710 P fcd494c8686d446b80b456554024d021 [term 1 FOLLOWER]: Refusing update from remote peer a03aec67e0484ff88f41cf7d400dac57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:05:23.041992 6119 raft_consensus.cc:1261] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 1 FOLLOWER]: Refusing update from remote peer a03aec67e0484ff88f41cf7d400dac57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:05:23.043143 6239 consensus_queue.cc:1035] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd494c8686d446b80b456554024d021" member_type: VOTER last_known_addr { host: "127.10.118.65" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:05:23.043820 6231 consensus_queue.cc:1035] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e73e6d9dd8514001b82505a7043afd41" member_type: VOTER last_known_addr { host: "127.10.118.67" port: 41863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:05:23.063633 6243 mvcc.cc:205] Tried to move back new op lower bound from 7317520068763045888 to 7317520068322975744. Current Snapshot: MvccSnapshot[applied={T|T < 7317520068763045888}]
I20260812 03:05:23.075814 6246 mvcc.cc:205] Tried to move back new op lower bound from 7317520068763045888 to 7317520068322975744. Current Snapshot: MvccSnapshot[applied={T|T < 7317520068763045888}]
I20260812 03:05:23.096261 5361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:05:23.096903 5361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 03:05:23.106920 5361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed364f4f72674976ba3e6146626778f5: Sending DeleteTablet for 3 replicas of tablet 51bac57af2434955a4277d6cf1661710
I20260812 03:05:23.108717 6073 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bac57af2434955a4277d6cf1661710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:23 UTC) from {username='slave'} at 127.0.0.1:32824
I20260812 03:05:23.109119 5833 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bac57af2434955a4277d6cf1661710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:23 UTC) from {username='slave'} at 127.0.0.1:53372
I20260812 03:05:23.109885 5583 tablet_service.cc:1514] Processing DeleteTablet for tablet 51bac57af2434955a4277d6cf1661710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 03:05:23 UTC) from {username='slave'} at 127.0.0.1:33144
I20260812 03:05:23.114905 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 5462
I20260812 03:05:23.133433 6264 tablet_replica.cc:330] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: stopping tablet replica
I20260812 03:05:23.134392 6264 raft_consensus.cc:2229] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 1 LEADER]: Raft consensus shutting down.
I20260812 03:05:23.135588 6264 raft_consensus.cc:2258] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:05:23.139489 6264 ts_tablet_manager.cc:1905] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:05:23.140990 6262 tablet_replica.cc:330] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: stopping tablet replica
I20260812 03:05:23.141902 6262 raft_consensus.cc:2229] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 03:05:23.142648 6262 raft_consensus.cc:2258] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 03:05:23.146091 6262 ts_tablet_manager.cc:1905] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 03:05:23.154896 6264 ts_tablet_manager.cc:1918] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:05:23.155365 6264 log.cc:1196] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-1/wal/wals/51bac57af2434955a4277d6cf1661710
I20260812 03:05:23.156440 6264 ts_tablet_manager.cc:1939] T 51bac57af2434955a4277d6cf1661710 P a03aec67e0484ff88f41cf7d400dac57: Deleting consensus metadata
I20260812 03:05:23.159782 5333 catalog_manager.cc:4841] TS a03aec67e0484ff88f41cf7d400dac57 (127.10.118.66:42959): tablet 51bac57af2434955a4277d6cf1661710 (table test-table [id=fba4f245999c4575870b05047d6aaba1]) successfully deleted
I20260812 03:05:23.161577 6262 ts_tablet_manager.cc:1918] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 03:05:23.161942 6262 log.cc:1196] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Deleting WAL directory at /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786503542438041-10713-0/minicluster-data/ts-2/wal/wals/51bac57af2434955a4277d6cf1661710
I20260812 03:05:23.162569 6262 ts_tablet_manager.cc:1939] T 51bac57af2434955a4277d6cf1661710 P e73e6d9dd8514001b82505a7043afd41: Deleting consensus metadata
I20260812 03:05:23.165021 5334 catalog_manager.cc:4841] TS e73e6d9dd8514001b82505a7043afd41 (127.10.118.67:41863): tablet 51bac57af2434955a4277d6cf1661710 (table test-table [id=fba4f245999c4575870b05047d6aaba1]) successfully deleted
W20260812 03:05:23.172013 5334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.118.65:37633 (error 108)
I20260812 03:05:23.173290 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 5724
I20260812 03:05:23.181711 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:23.179676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37633 (local address 127.0.0.1:33146)
0812 03:05:23.181005 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:23.181224 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":480,"threads_started":1}
W20260812 03:05:23.182138 5334 catalog_manager.cc:4584] TS fcd494c8686d446b80b456554024d021 (127.10.118.65:37633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51bac57af2434955a4277d6cf1661710: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37633: connect: Connection refused (error 111)
I20260812 03:05:23.207433 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 5961
I20260812 03:05:23.237519 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:23.236424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37633 (local address 127.0.0.1:33152)
0812 03:05:23.236931 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:23.237089 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 03:05:23.239778 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:23.238797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.65:37633 (local address 127.0.0.1:33160)
0812 03:05:23.239234 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0812 03:05:23.239377 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.65:37633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 03:05:23.255407 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 5298
2026-08-12T03:05:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (8313 ms)
[----------] 6 tests from EncryptionPolicyTest (55393 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:23 dist-test-slave-zpvx krb5kdc[6275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:23 dist-test-slave-zpvx krb5kdc[6275](info): set up 2 sockets
Aug 12 03:05:23 dist-test-slave-zpvx krb5kdc[6275](info): commencing operation
krb5kdc: starting...
W20260812 03:05:24.655052 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:24 dist-test-slave-zpvx krb5kdc[6275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:24.897029 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:25.205044 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:25.205662 6299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:25.206308 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:25.239312 6299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:25.239733 6299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:25.240046 6299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:25.240309 6299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:25.240621 6299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:25.277660 6299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:25.278954 6299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:25.280018 6299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:25.296890 6325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:25.297120 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:25.297132 6319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:25.298631 6299 server_base.cc:1034] running on GCE node
I20260812 03:05:26.482751 6299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:26.486119 6299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:26.487591 6299 hybrid_clock.cc:648] HybridClock initialized: now 1786503926487555 us; error 59 us; skew 500 ppm
I20260812 03:05:26.488581 6299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:26 dist-test-slave-zpvx krb5kdc[6275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:26.494745 6299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:26.500574 6299 webserver.cc:460] Webserver started at http://127.0.0.1:45415/ using document root <none> and password file <none>
I20260812 03:05:26.501508 6299 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:26.501744 6299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:26.502219 6299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:26.507439 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "e4772c131b0242eda3a2302b8e5a187e"
format_stamp: "Formatted at 2026-08-12 03:05:26 on dist-test-slave-zpvx"
server_key: "2b48218e9bb08b86ee6a59f8cad26124"
server_key_iv: "c30d696391f5c6c5c0f7409c1ed4412d"
server_key_version: "encryptionkey@0"
I20260812 03:05:26.508704 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "e4772c131b0242eda3a2302b8e5a187e"
format_stamp: "Formatted at 2026-08-12 03:05:26 on dist-test-slave-zpvx"
server_key: "2b48218e9bb08b86ee6a59f8cad26124"
server_key_iv: "c30d696391f5c6c5c0f7409c1ed4412d"
server_key_version: "encryptionkey@0"
I20260812 03:05:26.515980 6299 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:05:26.521533 6339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:26.522531 6299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 03:05:26.522876 6299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "e4772c131b0242eda3a2302b8e5a187e"
format_stamp: "Formatted at 2026-08-12 03:05:26 on dist-test-slave-zpvx"
server_key: "2b48218e9bb08b86ee6a59f8cad26124"
server_key_iv: "c30d696391f5c6c5c0f7409c1ed4412d"
server_key_version: "encryptionkey@0"
I20260812 03:05:26.523252 6299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:26.566872 6299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:26.572656 6299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:26.573129 6299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:26.644646 6299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39709
I20260812 03:05:26.644740 6427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39709 every 8 connection(s)
I20260812 03:05:26.647330 6299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:26.652571 6428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:26.653519 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 6299
I20260812 03:05:26.654009 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:26.655364 10713 external_mini_cluster.cc:1428] Setting key 01620ba4b19aa1acc44073d2e0f84b0e
Aug 12 03:05:26 dist-test-slave-zpvx krb5kdc[6275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 03:05:26.679183 6428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e: Bootstrap starting.
I20260812 03:05:26.686242 6428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:26.688091 6428 log.cc:824] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:26.694619 6428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e: No bootstrap required, opened a new log
I20260812 03:05:26.698792 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:26.657769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57785 (local address 127.0.0.1:39709)
0812 03:05:26.660255 (+ 2486us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:26.660294 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:26.660398 (+ 104us) server_negotiation.cc:407] Connection header received
0812 03:05:26.665071 (+ 4673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:26.665129 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:26.665639 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:26.666100 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:26.668632 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:26.671164 (+ 2532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:26.673206 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:26.674573 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:26.674900 (+ 327us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:26.680607 (+ 5707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:26.680639 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:26.680681 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:26.680824 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:26.686669 (+ 5845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:26.689417 (+ 2748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:26.689457 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:26.689490 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:26.689854 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:26.691100 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:26.691134 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:26.691153 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:26.692283 (+ 1130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:26.692855 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:26.695343 (+ 2488us) server_negotiation.cc:299] Negotiation successful
0812 03:05:26.695858 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1925,"thread_start_us":1544,"threads_started":1}
I20260812 03:05:26.703452 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:26.715144 6428 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4772c131b0242eda3a2302b8e5a187e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39709 } }
I20260812 03:05:26.716128 6428 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:26.716455 6428 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4772c131b0242eda3a2302b8e5a187e, State: Initialized, Role: FOLLOWER
I20260812 03:05:26.717299 6428 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4772c131b0242eda3a2302b8e5a187e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39709 } }
I20260812 03:05:26.717818 6428 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:26.718158 6428 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:26.718485 6428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:26.723480 6428 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4772c131b0242eda3a2302b8e5a187e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39709 } }
I20260812 03:05:26.724208 6428 leader_election.cc:302] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4772c131b0242eda3a2302b8e5a187e; no voters:
I20260812 03:05:26.726444 6428 leader_election.cc:288] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:26.726845 6446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:26.732800 6428 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:26.733927 6446 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [term 1 LEADER]: Becoming Leader. State: Replica: e4772c131b0242eda3a2302b8e5a187e, State: Running, Role: LEADER
I20260812 03:05:26.734763 6446 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4772c131b0242eda3a2302b8e5a187e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39709 } }
I20260812 03:05:26.747529 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:26.719618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38917 (local address 127.0.0.1:39709)
0812 03:05:26.720021 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:26.720039 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:26.721134 (+ 1095us) server_negotiation.cc:407] Connection header received
0812 03:05:26.721511 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:26.721543 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:26.721716 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:26.721882 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:26.724218 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:26.725312 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:26.727182 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:26.728469 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:26.728731 (+ 262us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:26.732231 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:26.732273 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:26.732298 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:26.732466 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:26.737288 (+ 4822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:26.739421 (+ 2133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:26.739451 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:26.739475 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:26.739742 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:26.741122 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:26.741215 (+ 93us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:26.741245 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:26.742167 (+ 922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:26.745391 (+ 3224us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:26.745651 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 03:05:26.745901 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 03:05:26.749478 6450 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4772c131b0242eda3a2302b8e5a187e. Latest consensus state: current_term: 1 leader_uuid: "e4772c131b0242eda3a2302b8e5a187e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4772c131b0242eda3a2302b8e5a187e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39709 } } }
I20260812 03:05:26.750183 6450 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:26.752429 6449 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4772c131b0242eda3a2302b8e5a187e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4772c131b0242eda3a2302b8e5a187e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39709 } } }
I20260812 03:05:26.753314 6449 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:26.754238 6462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:26.763273 6462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:26.783385 6462 catalog_manager.cc:1347] Generated new cluster ID: 98407c9295344648b00d7e768dba87a3
I20260812 03:05:26.784052 6462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:26.799394 6462 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:26.801118 6462 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:26.812544 6462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4772c131b0242eda3a2302b8e5a187e: Generated new TSK 0
I20260812 03:05:26.813560 6462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:26.825119 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 6299
W20260812 03:05:26.866230 6273 connection.cc:535] client connection to 127.0.0.1:39709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39709): Connection reset by peer (error 104)
2026-08-12T03:05:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (3565 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:26 dist-test-slave-zpvx krb5kdc[6490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:26 dist-test-slave-zpvx krb5kdc[6490](info): set up 2 sockets
Aug 12 03:05:26 dist-test-slave-zpvx krb5kdc[6490](info): commencing operation
krb5kdc: starting...
W20260812 03:05:28.218537 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.322s user 0.009s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:28 dist-test-slave-zpvx krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:28.453310 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:28.763944 6508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:28.764616 6508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:28.765297 6508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:28.796151 6508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:28.796586 6508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:28.796902 6508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:28.797173 6508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:28.797426 6508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:28.833448 6508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:28.834667 6508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:28.835711 6508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:28.853056 6521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:28.853053 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:28.854113 6523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:28.855577 6508 server_base.cc:1034] running on GCE node
I20260812 03:05:30.039202 6508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:30.042565 6508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:30.044006 6508 hybrid_clock.cc:648] HybridClock initialized: now 1786503930043950 us; error 79 us; skew 500 ppm
I20260812 03:05:30.044873 6508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:30 dist-test-slave-zpvx krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:30.056538 6508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:30.061853 6508 webserver.cc:460] Webserver started at http://127.0.0.1:46525/ using document root <none> and password file <none>
I20260812 03:05:30.062795 6508 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:30.063014 6508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:30.063534 6508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:30.068892 6508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "b3b58888cfdf41ab8a93dca57d985ac7"
format_stamp: "Formatted at 2026-08-12 03:05:30 on dist-test-slave-zpvx"
server_key: "90eaa45b2eb8fe1887569e0d5792ebe3"
server_key_iv: "b1aed506be04c23efef303ab4b7a4d72"
server_key_version: "encryptionkey@0"
I20260812 03:05:30.070024 6508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "b3b58888cfdf41ab8a93dca57d985ac7"
format_stamp: "Formatted at 2026-08-12 03:05:30 on dist-test-slave-zpvx"
server_key: "90eaa45b2eb8fe1887569e0d5792ebe3"
server_key_iv: "b1aed506be04c23efef303ab4b7a4d72"
server_key_version: "encryptionkey@0"
I20260812 03:05:30.077303 6508 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260812 03:05:30.082780 6537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:30.083853 6508 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:05:30.084213 6508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "b3b58888cfdf41ab8a93dca57d985ac7"
format_stamp: "Formatted at 2026-08-12 03:05:30 on dist-test-slave-zpvx"
server_key: "90eaa45b2eb8fe1887569e0d5792ebe3"
server_key_iv: "b1aed506be04c23efef303ab4b7a4d72"
server_key_version: "encryptionkey@0"
I20260812 03:05:30.084648 6508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:30.127161 6508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:30.132781 6508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:30.133265 6508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:30.219962 6508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45257
I20260812 03:05:30.220044 6613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45257 every 8 connection(s)
I20260812 03:05:30.222824 6508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:30.229544 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 6508
I20260812 03:05:30.230135 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:30.230710 6616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:30.231634 10713 external_mini_cluster.cc:1428] Setting key bac08e710492d432ad7cb4277db8c1c9
I20260812 03:05:30.261442 6616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7: Bootstrap starting.
I20260812 03:05:30.263305 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:30.234162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46065 (local address 127.0.0.1:45257)
0812 03:05:30.237295 (+ 3133us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:30.237327 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:30.237380 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:05:30.246839 (+ 9459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:30.246895 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:30.247390 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:30.249700 (+ 2310us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:30.250533 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:30.252415 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:30.255850 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:30.257082 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:30.257347 (+ 265us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:30.259515 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:30.259538 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:30.259571 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:30.260352 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:30.260540 (+ 188us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:30.260607 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:30.260789 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:30.261333 (+ 544us) server_negotiation.cc:299] Negotiation successful
0812 03:05:30.261534 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2712,"thread_start_us":2331,"threads_started":1}
I20260812 03:05:30.267988 6616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:30.268285 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:30.269810 6616 log.cc:824] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:30.274672 6616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7: No bootstrap required, opened a new log
I20260812 03:05:30.295295 6619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:30.282407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47039 (local address 127.0.0.1:45257)
0812 03:05:30.283103 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:30.283129 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:30.283969 (+ 840us) server_negotiation.cc:407] Connection header received
0812 03:05:30.284333 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:30.284355 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:30.284574 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:30.284715 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:30.285993 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:30.286966 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:30.289670 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:30.290467 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:30.290673 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:30.292723 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:30.292747 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:30.292764 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:30.292809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:30.292892 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:30.292944 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:30.293095 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:30.293601 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 03:05:30.293739 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260812 03:05:30.298298 6616 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260812 03:05:30.299022 6616 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:30.299281 6616 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b58888cfdf41ab8a93dca57d985ac7, State: Initialized, Role: FOLLOWER
I20260812 03:05:30.299973 6616 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260812 03:05:30.300598 6616 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:30.300879 6616 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:30.301187 6616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:30.305902 6616 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260812 03:05:30.306622 6616 leader_election.cc:302] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3b58888cfdf41ab8a93dca57d985ac7; no voters:
I20260812 03:05:30.308455 6616 leader_election.cc:288] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:30.308954 6630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:30.311506 6630 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [term 1 LEADER]: Becoming Leader. State: Replica: b3b58888cfdf41ab8a93dca57d985ac7, State: Running, Role: LEADER
I20260812 03:05:30.312302 6630 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } }
I20260812 03:05:30.313277 6616 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:30.322472 6637 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3b58888cfdf41ab8a93dca57d985ac7. Latest consensus state: current_term: 1 leader_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260812 03:05:30.322471 6634 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3b58888cfdf41ab8a93dca57d985ac7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45257 } } }
I20260812 03:05:30.323257 6637 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:30.323261 6634 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:30.332130 6651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:30.343056 6651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:30.359114 6651 catalog_manager.cc:1347] Generated new cluster ID: 5243b1422eb84ef9b4249a7b573b8f70
I20260812 03:05:30.359426 6651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:30.397946 6651 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:30.399686 6651 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:30.419796 6651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3b58888cfdf41ab8a93dca57d985ac7: Generated new TSK 0
I20260812 03:05:30.421072 6651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:30.445405 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 6508
W20260812 03:05:30.478493 6484 connection.cc:535] client connection to 127.0.0.1:45257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45257): Connection reset by peer (error 104)
2026-08-12T03:05:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (3609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:30 dist-test-slave-zpvx krb5kdc[6671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:30 dist-test-slave-zpvx krb5kdc[6671](info): set up 2 sockets
Aug 12 03:05:30 dist-test-slave-zpvx krb5kdc[6671](info): commencing operation
krb5kdc: starting...
W20260812 03:05:31.827715 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.324s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:31 dist-test-slave-zpvx krb5kdc[6671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:32.060612 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:32.367409 6691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:32.368045 6691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:32.368747 6691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:32.398928 6691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:32.399348 6691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:32.399665 6691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:32.399930 6691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:32.400189 6691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:32.436026 6691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:32.437327 6691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:32.438416 6691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:32.457211 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:32.458444 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:32.457255 6709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:33.634622 6708 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:05:33.634678 6691 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:05:33.639034 6691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:33.642257 6691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:33.643699 6691 hybrid_clock.cc:648] HybridClock initialized: now 1786503933643643 us; error 76 us; skew 500 ppm
I20260812 03:05:33.644582 6691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:33 dist-test-slave-zpvx krb5kdc[6671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:33.651170 6691 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:33.657037 6691 webserver.cc:460] Webserver started at http://127.0.0.1:37047/ using document root <none> and password file <none>
I20260812 03:05:33.657925 6691 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:33.658105 6691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:33.658516 6691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:33.663674 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "93518c4c42b14ac18851ce69ec5d2e3a"
format_stamp: "Formatted at 2026-08-12 03:05:33 on dist-test-slave-zpvx"
server_key: "dc17d5b87945d136374727cc1c355242"
server_key_iv: "d9dbe5e738a4a7318544bc6eb31378ae"
server_key_version: "encryptionkey@0"
I20260812 03:05:33.664878 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "93518c4c42b14ac18851ce69ec5d2e3a"
format_stamp: "Formatted at 2026-08-12 03:05:33 on dist-test-slave-zpvx"
server_key: "dc17d5b87945d136374727cc1c355242"
server_key_iv: "d9dbe5e738a4a7318544bc6eb31378ae"
server_key_version: "encryptionkey@0"
I20260812 03:05:33.672001 6691 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260812 03:05:33.677573 6727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:33.678579 6691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 03:05:33.678905 6691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "93518c4c42b14ac18851ce69ec5d2e3a"
format_stamp: "Formatted at 2026-08-12 03:05:33 on dist-test-slave-zpvx"
server_key: "dc17d5b87945d136374727cc1c355242"
server_key_iv: "d9dbe5e738a4a7318544bc6eb31378ae"
server_key_version: "encryptionkey@0"
I20260812 03:05:33.679273 6691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:33.705170 6691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:33.710791 6691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:33.711261 6691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:33.780256 6691 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43889
I20260812 03:05:33.780360 6809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43889 every 8 connection(s)
I20260812 03:05:33.783083 6691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:33.788348 6810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:33.793731 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 6691
I20260812 03:05:33.794430 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:33.795579 10713 external_mini_cluster.cc:1428] Setting key f63dff92536ffb1c1d6d0de6361f7868
I20260812 03:05:33.811285 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:33.798002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54079 (local address 127.0.0.1:43889)
0812 03:05:33.800674 (+ 2672us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:33.800713 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:33.800785 (+ 72us) server_negotiation.cc:407] Connection header received
0812 03:05:33.805576 (+ 4791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:33.805630 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:33.806041 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:33.806499 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:33.807991 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:33.808024 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:33.808063 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:33.808145 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:33.808301 (+ 156us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:33.808409 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:33.808709 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:33.808868 (+ 159us) server_negotiation.cc:299] Negotiation successful
0812 03:05:33.809103 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2130,"thread_start_us":1585,"threads_started":1}
I20260812 03:05:33.814419 6810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a: Bootstrap starting.
I20260812 03:05:33.816171 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:33.822018 6810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:33.824427 6810 log.cc:824] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:33.830513 6810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a: No bootstrap required, opened a new log
I20260812 03:05:33.842932 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:33.837499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50307 (local address 127.0.0.1:43889)
0812 03:05:33.837910 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:33.837929 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:33.838976 (+ 1047us) server_negotiation.cc:407] Connection header received
0812 03:05:33.839354 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:33.839373 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:33.839507 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:33.839671 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:33.841095 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:33.841116 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:33.841138 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:33.841179 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:33.841268 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:33.841317 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:33.841539 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:33.841797 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 03:05:33.841926 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 03:05:33.854954 6810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260812 03:05:33.855684 6810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:33.855947 6810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93518c4c42b14ac18851ce69ec5d2e3a, State: Initialized, Role: FOLLOWER
I20260812 03:05:33.856719 6810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260812 03:05:33.857251 6810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:33.857527 6810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:33.857846 6810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:33.862068 6810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260812 03:05:33.862891 6810 leader_election.cc:302] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93518c4c42b14ac18851ce69ec5d2e3a; no voters:
I20260812 03:05:33.865118 6810 leader_election.cc:288] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:33.865535 6827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:33.868026 6827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [term 1 LEADER]: Becoming Leader. State: Replica: 93518c4c42b14ac18851ce69ec5d2e3a, State: Running, Role: LEADER
I20260812 03:05:33.868958 6827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260812 03:05:33.870357 6810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:33.880479 6831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93518c4c42b14ac18851ce69ec5d2e3a. Latest consensus state: current_term: 1 leader_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } } }
I20260812 03:05:33.881417 6831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:33.883930 6830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93518c4c42b14ac18851ce69ec5d2e3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } } }
I20260812 03:05:33.884799 6830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:33.885056 6847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:33.895217 6847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:33.913605 6847 catalog_manager.cc:1347] Generated new cluster ID: 2fea34ec9c0340b0b6db8eb9f7d19afa
I20260812 03:05:33.913921 6847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:33.938755 6847 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:33.939389 6847 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:33.961127 6847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93518c4c42b14ac18851ce69ec5d2e3a: Generated new TSK 0
I20260812 03:05:33.962281 6847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:33.972002 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 6691
2026-08-12T03:05:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (3530 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:34 dist-test-slave-zpvx krb5kdc[6874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:34 dist-test-slave-zpvx krb5kdc[6874](info): set up 2 sockets
Aug 12 03:05:34 dist-test-slave-zpvx krb5kdc[6874](info): commencing operation
krb5kdc: starting...
W20260812 03:05:35.348850 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:35 dist-test-slave-zpvx krb5kdc[6874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:35.586920 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:35.894897 6893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:35.895533 6893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:35.896173 6893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:35.926455 6893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:35.926877 6893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:35.927191 6893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:35.927460 6893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:35.927709 6893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:35.963090 6893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:35.964305 6893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:35.965353 6893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:35.983502 6908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:35.983515 6906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:35.984491 6912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:35.985244 6893 server_base.cc:1034] running on GCE node
I20260812 03:05:37.173791 6893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:37.177026 6893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:37.178459 6893 hybrid_clock.cc:648] HybridClock initialized: now 1786503937178431 us; error 46 us; skew 500 ppm
I20260812 03:05:37.179294 6893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:37 dist-test-slave-zpvx krb5kdc[6874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:37.190165 6893 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:37.195991 6893 webserver.cc:460] Webserver started at http://127.0.0.1:34185/ using document root <none> and password file <none>
I20260812 03:05:37.196996 6893 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:37.197244 6893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:37.197717 6893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:37.202910 6893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "e61766936ec44137b5cda82cc8f77a12"
format_stamp: "Formatted at 2026-08-12 03:05:37 on dist-test-slave-zpvx"
server_key: "4e19d19ce189b8539a63d194a892baaa"
server_key_iv: "38b820096c42c2cd45a349cd3b617709"
server_key_version: "encryptionkey@0"
I20260812 03:05:37.204075 6893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "e61766936ec44137b5cda82cc8f77a12"
format_stamp: "Formatted at 2026-08-12 03:05:37 on dist-test-slave-zpvx"
server_key: "4e19d19ce189b8539a63d194a892baaa"
server_key_iv: "38b820096c42c2cd45a349cd3b617709"
server_key_version: "encryptionkey@0"
I20260812 03:05:37.211611 6893 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 03:05:37.217222 6922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:37.218245 6893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 03:05:37.218580 6893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "e61766936ec44137b5cda82cc8f77a12"
format_stamp: "Formatted at 2026-08-12 03:05:37 on dist-test-slave-zpvx"
server_key: "4e19d19ce189b8539a63d194a892baaa"
server_key_iv: "38b820096c42c2cd45a349cd3b617709"
server_key_version: "encryptionkey@0"
I20260812 03:05:37.218973 6893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:37.255159 6893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:37.260978 6893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:37.261457 6893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:37.334288 6893 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45651
I20260812 03:05:37.334367 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45651 every 8 connection(s)
I20260812 03:05:37.336997 6893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:37.342315 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:37.344687 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 6893
I20260812 03:05:37.345142 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:37.346169 10713 external_mini_cluster.cc:1428] Setting key 6433fbb6cba39279b049fbbe82b89080
I20260812 03:05:37.363502 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12: Bootstrap starting.
I20260812 03:05:37.370687 7016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12: Neither blocks nor log segments found. Creating new log.
Aug 12 03:05:37 dist-test-slave-zpvx krb5kdc[6874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 03:05:37.372560 7016 log.cc:824] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:37.378763 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12: No bootstrap required, opened a new log
I20260812 03:05:37.385121 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:37.348538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47429 (local address 127.0.0.1:45651)
0812 03:05:37.351249 (+ 2711us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:37.351282 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:37.351332 (+ 50us) server_negotiation.cc:407] Connection header received
0812 03:05:37.355217 (+ 3885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:37.355259 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:37.355610 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:37.355918 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:37.359727 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:37.361518 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:37.364301 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:37.365183 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:37.365420 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:37.372520 (+ 7100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:37.372562 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:37.372602 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:37.372762 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:37.378349 (+ 5587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:37.379594 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:37.379617 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:37.379634 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:37.379887 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:37.380617 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:37.380652 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:37.380678 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:37.381400 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:37.381872 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:37.382572 (+ 700us) server_negotiation.cc:299] Negotiation successful
0812 03:05:37.383203 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2293,"thread_start_us":1812,"threads_started":1}
I20260812 03:05:37.389338 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:37.399147 7016 raft_consensus.cc:348] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61766936ec44137b5cda82cc8f77a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45651 } }
I20260812 03:05:37.399829 7016 raft_consensus.cc:374] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:37.400091 7016 raft_consensus.cc:729] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61766936ec44137b5cda82cc8f77a12, State: Initialized, Role: FOLLOWER
I20260812 03:05:37.400895 7016 consensus_queue.cc:260] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61766936ec44137b5cda82cc8f77a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45651 } }
I20260812 03:05:37.401434 7016 raft_consensus.cc:388] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:37.401758 7016 raft_consensus.cc:482] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:37.402115 7016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:37.407004 7016 raft_consensus.cc:504] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61766936ec44137b5cda82cc8f77a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45651 } }
I20260812 03:05:37.407848 7016 leader_election.cc:302] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e61766936ec44137b5cda82cc8f77a12; no voters:
I20260812 03:05:37.409780 7016 leader_election.cc:288] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:37.410228 7039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:37.412524 7039 raft_consensus.cc:686] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [term 1 LEADER]: Becoming Leader. State: Replica: e61766936ec44137b5cda82cc8f77a12, State: Running, Role: LEADER
I20260812 03:05:37.413340 7039 consensus_queue.cc:237] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61766936ec44137b5cda82cc8f77a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45651 } }
I20260812 03:05:37.416973 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61766936ec44137b5cda82cc8f77a12. Latest consensus state: current_term: 1 leader_uuid: "e61766936ec44137b5cda82cc8f77a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61766936ec44137b5cda82cc8f77a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45651 } } }
I20260812 03:05:37.417789 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:37.422933 7016 sys_catalog.cc:564] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:37.424077 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61766936ec44137b5cda82cc8f77a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e61766936ec44137b5cda82cc8f77a12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45651 } } }
I20260812 03:05:37.424875 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:37.428146 7050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:37.437762 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:37.401945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40841 (local address 127.0.0.1:45651)
0812 03:05:37.402447 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:37.402472 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:37.403589 (+ 1117us) server_negotiation.cc:407] Connection header received
0812 03:05:37.403870 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:37.403902 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:37.404137 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:37.404323 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:37.406467 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:37.407873 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:37.410444 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:37.411615 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:37.413572 (+ 1957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:37.418797 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:37.418848 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:37.418903 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:37.419141 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:37.428180 (+ 9039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:37.430075 (+ 1895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:37.430124 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:37.430163 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:37.430458 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:37.431363 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:37.431399 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:37.431434 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:37.432276 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:37.432789 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:37.434214 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0812 03:05:37.434826 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 03:05:37.443060 7050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:37.462558 7050 catalog_manager.cc:1347] Generated new cluster ID: 0a679ec40fda4a4f8606e59c927c3d0f
I20260812 03:05:37.462899 7050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:37.508630 7050 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:37.511272 7050 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:37.532385 7050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e61766936ec44137b5cda82cc8f77a12: Generated new TSK 0
I20260812 03:05:37.533322 7050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:37.544322 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 6893
2026-08-12T03:05:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (3571 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:37 dist-test-slave-zpvx krb5kdc[7076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:37 dist-test-slave-zpvx krb5kdc[7076](info): set up 2 sockets
Aug 12 03:05:37 dist-test-slave-zpvx krb5kdc[7076](info): commencing operation
krb5kdc: starting...
W20260812 03:05:38.918732 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:39 dist-test-slave-zpvx krb5kdc[7076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:39.148176 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:39.456441 7104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:39.457090 7104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:39.457736 7104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:39.488437 7104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:39.488857 7104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:39.489208 7104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:39.489475 7104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:39.489730 7104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:39.526464 7104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:39.527695 7104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:39.528781 7104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:39.545814 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:39.545819 7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:39.545892 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:40.748693 7118 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:05:40.748826 7104 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:05:40.752243 7104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:40.754906 7104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:40.756300 7104 hybrid_clock.cc:648] HybridClock initialized: now 1786503940756262 us; error 71 us; skew 500 ppm
I20260812 03:05:40.757186 7104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:40 dist-test-slave-zpvx krb5kdc[7076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:40.763227 7104 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:40.768692 7104 webserver.cc:460] Webserver started at http://127.0.0.1:33917/ using document root <none> and password file <none>
I20260812 03:05:40.769636 7104 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:40.769868 7104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:40.770351 7104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:40.775653 7104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "4df7782a63484ae8ba637b25355569d2"
format_stamp: "Formatted at 2026-08-12 03:05:40 on dist-test-slave-zpvx"
server_key: "716f39d10a61f11071bfaffd842d3283"
server_key_iv: "ec2ea1ef183692dde15966c20d187506"
server_key_version: "encryptionkey@0"
I20260812 03:05:40.776934 7104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "4df7782a63484ae8ba637b25355569d2"
format_stamp: "Formatted at 2026-08-12 03:05:40 on dist-test-slave-zpvx"
server_key: "716f39d10a61f11071bfaffd842d3283"
server_key_iv: "ec2ea1ef183692dde15966c20d187506"
server_key_version: "encryptionkey@0"
I20260812 03:05:40.784210 7104 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:05:40.789947 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:40.791117 7104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260812 03:05:40.791515 7104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "4df7782a63484ae8ba637b25355569d2"
format_stamp: "Formatted at 2026-08-12 03:05:40 on dist-test-slave-zpvx"
server_key: "716f39d10a61f11071bfaffd842d3283"
server_key_iv: "ec2ea1ef183692dde15966c20d187506"
server_key_version: "encryptionkey@0"
I20260812 03:05:40.791914 7104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:40.847436 7104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:40.853505 7104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:40.854003 7104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:40.923336 7104 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35537
I20260812 03:05:40.923413 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35537 every 8 connection(s)
I20260812 03:05:40.926041 7104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:40.931329 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:40.933315 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 7104
I20260812 03:05:40.933907 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:40.935415 10713 external_mini_cluster.cc:1428] Setting key 5b4513fb204bdb3a5b9585d7ae0718a9
I20260812 03:05:40.958491 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2: Bootstrap starting.
I20260812 03:05:40.958738 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:40.937985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48205 (local address 127.0.0.1:35537)
0812 03:05:40.940786 (+ 2801us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:40.940813 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:40.940867 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:05:40.944794 (+ 3927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:40.944851 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:40.945219 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:40.945544 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:40.947021 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:40.948963 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:40.951991 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:40.952931 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:40.953170 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:40.955131 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:40.955154 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:40.955183 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:40.955932 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:40.956052 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:40.956110 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:40.956233 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:40.956680 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 03:05:40.956931 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2356,"thread_start_us":1865,"threads_started":1}
I20260812 03:05:40.963552 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:40.965348 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:40.967044 7223 log.cc:824] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:40.971927 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2: No bootstrap required, opened a new log
I20260812 03:05:40.989491 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:40.977473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44335 (local address 127.0.0.1:35537)
0812 03:05:40.978177 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:40.978226 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:40.979110 (+ 884us) server_negotiation.cc:407] Connection header received
0812 03:05:40.979486 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:40.979504 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:40.979671 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:40.979795 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:40.981143 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:40.981979 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:40.984654 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:40.985218 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:40.985417 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:40.987337 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:40.987357 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:40.987379 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:40.987420 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:40.987503 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:40.987550 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:40.987657 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:40.988003 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 03:05:40.988150 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260812 03:05:40.995682 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df7782a63484ae8ba637b25355569d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260812 03:05:40.996426 7223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:40.996692 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df7782a63484ae8ba637b25355569d2, State: Initialized, Role: FOLLOWER
I20260812 03:05:40.997470 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df7782a63484ae8ba637b25355569d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260812 03:05:40.998191 7223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:40.998509 7223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:40.998835 7223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:41.003053 7223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df7782a63484ae8ba637b25355569d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260812 03:05:41.003944 7223 leader_election.cc:302] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df7782a63484ae8ba637b25355569d2; no voters:
I20260812 03:05:41.005829 7223 leader_election.cc:288] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:41.006254 7241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:41.008891 7241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4df7782a63484ae8ba637b25355569d2, State: Running, Role: LEADER
I20260812 03:05:41.009682 7241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df7782a63484ae8ba637b25355569d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } }
I20260812 03:05:41.010645 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:41.022339 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df7782a63484ae8ba637b25355569d2. Latest consensus state: current_term: 1 leader_uuid: "4df7782a63484ae8ba637b25355569d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df7782a63484ae8ba637b25355569d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260812 03:05:41.022635 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df7782a63484ae8ba637b25355569d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df7782a63484ae8ba637b25355569d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35537 } } }
I20260812 03:05:41.023146 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:41.023311 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:41.025784 7262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:41.038889 7262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:41.056227 7262 catalog_manager.cc:1347] Generated new cluster ID: ac9c13d2f3554a778492f206d644c993
I20260812 03:05:41.056569 7262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:41.085675 7262 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:41.087510 7262 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:41.109925 7262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df7782a63484ae8ba637b25355569d2: Generated new TSK 0
I20260812 03:05:41.110854 7262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:41.128242 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 7104
2026-08-12T03:05:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (3583 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:41 dist-test-slave-zpvx krb5kdc[7282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:41 dist-test-slave-zpvx krb5kdc[7282](info): set up 2 sockets
Aug 12 03:05:41 dist-test-slave-zpvx krb5kdc[7282](info): commencing operation
krb5kdc: starting...
W20260812 03:05:42.503867 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:42 dist-test-slave-zpvx krb5kdc[7282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:42.748749 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:43.058876 7300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:43.059547 7300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:43.060210 7300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:43.090924 7300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:43.091353 7300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:43.091665 7300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:43.091933 7300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:43.092206 7300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:43.128196 7300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:43.129530 7300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:43.130650 7300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 03:05:43.150024 7300 server_base.cc:1034] running on GCE node
W20260812 03:05:43.148914 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:43.148293 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:43.148293 7315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:44.349694 7300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:44.352945 7300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:44.354382 7300 hybrid_clock.cc:648] HybridClock initialized: now 1786503944354323 us; error 82 us; skew 500 ppm
I20260812 03:05:44.355157 7300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:44 dist-test-slave-zpvx krb5kdc[7282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:44.367036 7300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:44.372517 7300 webserver.cc:460] Webserver started at http://127.0.0.1:34155/ using document root <none> and password file <none>
I20260812 03:05:44.373466 7300 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:44.373714 7300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:44.374207 7300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:44.379482 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "811ab9e9027d4dacbf1a453ce61e56f4"
format_stamp: "Formatted at 2026-08-12 03:05:44 on dist-test-slave-zpvx"
server_key: "1b3e705192fc8a6a6d1754b66b23a21b"
server_key_iv: "2b620bbac1adf8ba0b9fb804071b4e9b"
server_key_version: "encryptionkey@0"
I20260812 03:05:44.380752 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "811ab9e9027d4dacbf1a453ce61e56f4"
format_stamp: "Formatted at 2026-08-12 03:05:44 on dist-test-slave-zpvx"
server_key: "1b3e705192fc8a6a6d1754b66b23a21b"
server_key_iv: "2b620bbac1adf8ba0b9fb804071b4e9b"
server_key_version: "encryptionkey@0"
I20260812 03:05:44.388020 7300 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 03:05:44.393646 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:44.394699 7300 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 03:05:44.395026 7300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "811ab9e9027d4dacbf1a453ce61e56f4"
format_stamp: "Formatted at 2026-08-12 03:05:44 on dist-test-slave-zpvx"
server_key: "1b3e705192fc8a6a6d1754b66b23a21b"
server_key_iv: "2b620bbac1adf8ba0b9fb804071b4e9b"
server_key_version: "encryptionkey@0"
I20260812 03:05:44.395431 7300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:44.420707 7300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:44.426229 7300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:44.426715 7300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:44.495886 7300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45175
I20260812 03:05:44.495981 7439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45175 every 8 connection(s)
I20260812 03:05:44.498777 7300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:44.504047 7441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:44.508752 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 7300
I20260812 03:05:44.509657 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:44.511340 10713 external_mini_cluster.cc:1428] Setting key 31145a7bb8d6a040473d7e9c41098831
I20260812 03:05:44.526641 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:44.514273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35447 (local address 127.0.0.1:45175)
0812 03:05:44.517185 (+ 2912us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:44.517214 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:44.517272 (+ 58us) server_negotiation.cc:407] Connection header received
0812 03:05:44.521346 (+ 4074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:44.521384 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:44.521685 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:44.522008 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:44.523825 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:44.523848 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:44.523875 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:44.523920 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:44.524026 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:44.524133 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:44.524355 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:44.524689 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 03:05:44.524858 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2461,"thread_start_us":1926,"threads_started":1}
I20260812 03:05:44.528854 7441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4: Bootstrap starting.
I20260812 03:05:44.531563 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:44.535622 7441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:44.538076 7441 log.cc:824] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:44.544323 7441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4: No bootstrap required, opened a new log
I20260812 03:05:44.559392 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:44.553259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52909 (local address 127.0.0.1:45175)
0812 03:05:44.553800 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:44.553827 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:44.554886 (+ 1059us) server_negotiation.cc:407] Connection header received
0812 03:05:44.555208 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:44.555239 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:44.555421 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:44.555633 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:44.557032 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:44.557062 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:44.557093 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:44.557153 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:44.557269 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:44.557337 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:44.557587 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:44.557828 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 03:05:44.557954 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 03:05:44.569628 7441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260812 03:05:44.570313 7441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:44.570515 7441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811ab9e9027d4dacbf1a453ce61e56f4, State: Initialized, Role: FOLLOWER
I20260812 03:05:44.571214 7441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260812 03:05:44.571697 7441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:44.571919 7441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:44.572165 7441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:44.576828 7441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260812 03:05:44.577504 7441 leader_election.cc:302] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 811ab9e9027d4dacbf1a453ce61e56f4; no voters:
I20260812 03:05:44.579109 7441 leader_election.cc:288] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:44.579671 7464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:44.582558 7464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [term 1 LEADER]: Becoming Leader. State: Replica: 811ab9e9027d4dacbf1a453ce61e56f4, State: Running, Role: LEADER
I20260812 03:05:44.583256 7464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } }
I20260812 03:05:44.583859 7441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:44.593811 7466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 811ab9e9027d4dacbf1a453ce61e56f4. Latest consensus state: current_term: 1 leader_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } } }
I20260812 03:05:44.595062 7465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811ab9e9027d4dacbf1a453ce61e56f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45175 } } }
I20260812 03:05:44.595853 7465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:44.595813 7466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:44.600473 7475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:44.611969 7475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:44.628588 7475 catalog_manager.cc:1347] Generated new cluster ID: cfc098881f9442c8a093ce91ee392135
I20260812 03:05:44.628906 7475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:44.649294 7475 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:44.649902 7475 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:44.664350 7475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 811ab9e9027d4dacbf1a453ce61e56f4: Generated new TSK 0
I20260812 03:05:44.665463 7475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:44.683176 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 7300
2026-08-12T03:05:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (3562 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:44 dist-test-slave-zpvx krb5kdc[7489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:44 dist-test-slave-zpvx krb5kdc[7489](info): set up 2 sockets
Aug 12 03:05:44 dist-test-slave-zpvx krb5kdc[7489](info): commencing operation
krb5kdc: starting...
W20260812 03:05:46.068754 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.316s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:46 dist-test-slave-zpvx krb5kdc[7489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:46.304783 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:46.612839 7509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:46.613425 7509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:46.614001 7509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:46.644315 7509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:46.644654 7509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:46.644999 7509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:46.645223 7509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:46.645411 7509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:46.680819 7509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:46.681946 7509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:46.682931 7509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:46.704185 7527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:46.705180 7524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:46.704173 7522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:47.891433 7525 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:05:47.891551 7509 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:05:47.896071 7509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:47.899682 7509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:47.901180 7509 hybrid_clock.cc:648] HybridClock initialized: now 1786503947901130 us; error 71 us; skew 500 ppm
I20260812 03:05:47.902107 7509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:47 dist-test-slave-zpvx krb5kdc[7489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:47.908921 7509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:47.915277 7509 webserver.cc:460] Webserver started at http://127.0.0.1:35433/ using document root <none> and password file <none>
I20260812 03:05:47.916301 7509 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:47.916651 7509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:47.917223 7509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:47.923156 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "1ca426c204494c34ba7816534d5691f5"
format_stamp: "Formatted at 2026-08-12 03:05:47 on dist-test-slave-zpvx"
server_key: "95974b79e2ee413fa409260074ffe696"
server_key_iv: "a10478c87192b3298e45d52ff23a0808"
server_key_version: "encryptionkey@0"
I20260812 03:05:47.924568 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "1ca426c204494c34ba7816534d5691f5"
format_stamp: "Formatted at 2026-08-12 03:05:47 on dist-test-slave-zpvx"
server_key: "95974b79e2ee413fa409260074ffe696"
server_key_iv: "a10478c87192b3298e45d52ff23a0808"
server_key_version: "encryptionkey@0"
I20260812 03:05:47.932761 7509 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 03:05:47.939021 7539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:47.940101 7509 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 03:05:47.940454 7509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "1ca426c204494c34ba7816534d5691f5"
format_stamp: "Formatted at 2026-08-12 03:05:47 on dist-test-slave-zpvx"
server_key: "95974b79e2ee413fa409260074ffe696"
server_key_iv: "a10478c87192b3298e45d52ff23a0808"
server_key_version: "encryptionkey@0"
I20260812 03:05:47.940888 7509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:47.981906 7509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:47.987839 7509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:47.988395 7509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:48.059998 7509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36667
I20260812 03:05:48.060068 7642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36667 every 8 connection(s)
I20260812 03:05:48.062882 7509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:48.068591 7644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:48.070524 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 7509
I20260812 03:05:48.071009 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:48.072041 10713 external_mini_cluster.cc:1428] Setting key bfbd6153c8c46b158e230c2a5ed5ccbc
I20260812 03:05:48.095690 7644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5: Bootstrap starting.
Aug 12 03:05:48 dist-test-slave-zpvx krb5kdc[7489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 03:05:48.102078 7644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:48.104010 7644 log.cc:824] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:48.110456 7644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5: No bootstrap required, opened a new log
I20260812 03:05:48.111235 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:48.074485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:43087 (local address 127.0.0.1:36667)
0812 03:05:48.077282 (+ 2797us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:48.077309 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:48.077362 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:05:48.081310 (+ 3948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:48.081355 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:48.081713 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:48.082035 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:48.085583 (+ 3548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:48.087227 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:48.090198 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:48.091075 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:48.091309 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:48.098734 (+ 7425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:48.098764 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:48.098792 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:48.098894 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:48.103242 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:48.104957 (+ 1715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:48.104978 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:48.104995 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:48.105234 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:48.106357 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:48.106391 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:48.106420 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:48.107547 (+ 1127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:48.108117 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:48.108769 (+ 652us) server_negotiation.cc:299] Negotiation successful
0812 03:05:48.109134 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2359,"thread_start_us":1872,"threads_started":1}
I20260812 03:05:48.115959 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:48.131261 7644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca426c204494c34ba7816534d5691f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260812 03:05:48.131976 7644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:48.132236 7644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ca426c204494c34ba7816534d5691f5, State: Initialized, Role: FOLLOWER
I20260812 03:05:48.132964 7644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca426c204494c34ba7816534d5691f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260812 03:05:48.133543 7644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:48.133920 7644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:48.134320 7644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:48.139385 7644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca426c204494c34ba7816534d5691f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260812 03:05:48.140125 7644 leader_election.cc:302] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ca426c204494c34ba7816534d5691f5; no voters:
I20260812 03:05:48.142416 7644 leader_election.cc:288] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:48.142863 7660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:48.146040 7660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [term 1 LEADER]: Becoming Leader. State: Replica: 1ca426c204494c34ba7816534d5691f5, State: Running, Role: LEADER
I20260812 03:05:48.146873 7660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca426c204494c34ba7816534d5691f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } }
I20260812 03:05:48.147747 7644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:48.158010 7649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:48.128672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:43613 (local address 127.0.0.1:36667)
0812 03:05:48.129217 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:48.129246 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:48.130199 (+ 953us) server_negotiation.cc:407] Connection header received
0812 03:05:48.130492 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:48.130522 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:48.130756 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:48.130936 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:48.133468 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:48.135016 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:48.136863 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:48.137812 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:48.138092 (+ 280us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:48.141981 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:48.142016 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:48.142048 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:48.142206 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:48.147127 (+ 4921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:48.148554 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:48.148574 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:48.148588 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:48.148778 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:48.149924 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:48.149981 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:48.150006 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:48.150834 (+ 828us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:48.151449 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:48.154241 (+ 2792us) server_negotiation.cc:299] Negotiation successful
0812 03:05:48.154756 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":104}
I20260812 03:05:48.158720 7667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ca426c204494c34ba7816534d5691f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca426c204494c34ba7816534d5691f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } } }
I20260812 03:05:48.159734 7667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:48.158046 7668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ca426c204494c34ba7816534d5691f5. Latest consensus state: current_term: 1 leader_uuid: "1ca426c204494c34ba7816534d5691f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ca426c204494c34ba7816534d5691f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36667 } } }
I20260812 03:05:48.163282 7668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:48.166950 7680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:48.178774 7680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:48.195957 7680 catalog_manager.cc:1347] Generated new cluster ID: 1d6cb7ffab1549ffaf13c04a0499c7b1
I20260812 03:05:48.196266 7680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:48.242105 7680 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:48.244578 7680 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:48.262223 7680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ca426c204494c34ba7816534d5691f5: Generated new TSK 0
I20260812 03:05:48.263386 7680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:48.278156 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 7509
W20260812 03:05:48.314569 7486 connection.cc:535] client connection to 127.0.0.1:36667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36667): Connection reset by peer (error 104)
2026-08-12T03:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (3588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:48 dist-test-slave-zpvx krb5kdc[7703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:48 dist-test-slave-zpvx krb5kdc[7703](info): set up 2 sockets
Aug 12 03:05:48 dist-test-slave-zpvx krb5kdc[7703](info): commencing operation
krb5kdc: starting...
W20260812 03:05:49.717742 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.377s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:49 dist-test-slave-zpvx krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:49.952138 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:50.257928 7726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:50.258490 7726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:50.259054 7726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:50.289618 7726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:50.289942 7726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:50.290184 7726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:50.290380 7726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:50.290561 7726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:50.334936 7726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:50.336426 7726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:50.337992 7726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:50.358825 7741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:50.359083 7746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:50.360103 7740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:50.360452 7726 server_base.cc:1034] running on GCE node
I20260812 03:05:51.548344 7726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:51.551671 7726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:51.553098 7726 hybrid_clock.cc:648] HybridClock initialized: now 1786503951553039 us; error 79 us; skew 500 ppm
I20260812 03:05:51.553864 7726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:51 dist-test-slave-zpvx krb5kdc[7703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:51.559850 7726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:51.565236 7726 webserver.cc:460] Webserver started at http://127.0.0.1:43487/ using document root <none> and password file <none>
I20260812 03:05:51.566177 7726 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:51.566421 7726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:51.566895 7726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:51.572678 7726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "9949310ce74c43c1a5b351b04d831187"
format_stamp: "Formatted at 2026-08-12 03:05:51 on dist-test-slave-zpvx"
server_key: "9883a9f8e85fa56add85afdd1e977276"
server_key_iv: "de0a0156f03222040470d23c71cf1f85"
server_key_version: "encryptionkey@0"
I20260812 03:05:51.573873 7726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "9949310ce74c43c1a5b351b04d831187"
format_stamp: "Formatted at 2026-08-12 03:05:51 on dist-test-slave-zpvx"
server_key: "9883a9f8e85fa56add85afdd1e977276"
server_key_iv: "de0a0156f03222040470d23c71cf1f85"
server_key_version: "encryptionkey@0"
I20260812 03:05:51.581259 7726 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:05:51.586793 7758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:51.587772 7726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 03:05:51.588110 7726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "9949310ce74c43c1a5b351b04d831187"
format_stamp: "Formatted at 2026-08-12 03:05:51 on dist-test-slave-zpvx"
server_key: "9883a9f8e85fa56add85afdd1e977276"
server_key_iv: "de0a0156f03222040470d23c71cf1f85"
server_key_version: "encryptionkey@0"
I20260812 03:05:51.588553 7726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:51.624022 7726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:51.629485 7726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:51.629966 7726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:51.698664 7726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37671
I20260812 03:05:51.698751 7860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37671 every 8 connection(s)
I20260812 03:05:51.701421 7726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:51.707440 7864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:51.708953 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 7726
I20260812 03:05:51.709527 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:51.710597 10713 external_mini_cluster.cc:1428] Setting key b2a983d2c2758f40f7af85f734bd585c
I20260812 03:05:51.732342 7864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187: Bootstrap starting.
I20260812 03:05:51.734035 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:51.712763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:55427 (local address 127.0.0.1:37671)
0812 03:05:51.715337 (+ 2574us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:51.715375 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:51.715449 (+ 74us) server_negotiation.cc:407] Connection header received
0812 03:05:51.720755 (+ 5306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:51.720811 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:51.721319 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:51.721773 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:51.722877 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:51.724564 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:51.726643 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:51.727472 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:51.727826 (+ 354us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:51.729841 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:51.729863 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:51.729893 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:51.730616 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:51.730738 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:51.730801 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:51.731051 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:51.731512 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 03:05:51.731727 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2017,"thread_start_us":1622,"threads_started":1}
I20260812 03:05:51.739230 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:51.739594 7864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:51.741396 7864 log.cc:824] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:51.746309 7864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187: No bootstrap required, opened a new log
I20260812 03:05:51.764147 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:51.753314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:38659 (local address 127.0.0.1:37671)
0812 03:05:51.753721 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:51.753745 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:51.754766 (+ 1021us) server_negotiation.cc:407] Connection header received
0812 03:05:51.755146 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:51.755168 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:51.755331 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:51.755497 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:51.756535 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:51.757511 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:51.759338 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:51.759913 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:51.760202 (+ 289us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:51.762022 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:51.762043 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:51.762059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:51.762116 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:51.762202 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:51.762253 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:51.762465 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:51.762887 (+ 422us) server_negotiation.cc:299] Negotiation successful
0812 03:05:51.763007 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 03:05:51.768744 7864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9949310ce74c43c1a5b351b04d831187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260812 03:05:51.769410 7864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:51.769706 7864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9949310ce74c43c1a5b351b04d831187, State: Initialized, Role: FOLLOWER
I20260812 03:05:51.770435 7864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9949310ce74c43c1a5b351b04d831187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260812 03:05:51.770944 7864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:51.771251 7864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:51.771636 7864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:51.776608 7864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9949310ce74c43c1a5b351b04d831187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260812 03:05:51.777526 7864 leader_election.cc:302] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9949310ce74c43c1a5b351b04d831187; no voters:
I20260812 03:05:51.779269 7864 leader_election.cc:288] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:51.779683 7887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:51.782070 7887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [term 1 LEADER]: Becoming Leader. State: Replica: 9949310ce74c43c1a5b351b04d831187, State: Running, Role: LEADER
I20260812 03:05:51.783672 7887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9949310ce74c43c1a5b351b04d831187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } }
I20260812 03:05:51.784083 7864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:51.796190 7888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9949310ce74c43c1a5b351b04d831187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9949310ce74c43c1a5b351b04d831187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } } }
I20260812 03:05:51.796356 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9949310ce74c43c1a5b351b04d831187. Latest consensus state: current_term: 1 leader_uuid: "9949310ce74c43c1a5b351b04d831187" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9949310ce74c43c1a5b351b04d831187" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37671 } } }
I20260812 03:05:51.797348 7888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:51.797433 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:51.801061 7901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:51.814102 7901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:51.832084 7901 catalog_manager.cc:1347] Generated new cluster ID: 7f5c844bde41480b87e8a450b8a14f41
I20260812 03:05:51.832404 7901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:51.849469 7901 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:51.851222 7901 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:51.864292 7901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9949310ce74c43c1a5b351b04d831187: Generated new TSK 0
I20260812 03:05:51.865278 7901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:51.882503 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 7726
W20260812 03:05:51.913796 7700 connection.cc:535] client connection to 127.0.0.1:37671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37671): Connection reset by peer (error 104)
2026-08-12T03:05:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (3598 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:52 dist-test-slave-zpvx krb5kdc[7928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:52 dist-test-slave-zpvx krb5kdc[7928](info): set up 2 sockets
Aug 12 03:05:52 dist-test-slave-zpvx krb5kdc[7928](info): commencing operation
krb5kdc: starting...
W20260812 03:05:53.270327 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:53 dist-test-slave-zpvx krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:53.530043 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:53.839273 7950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:53.839905 7950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:53.840672 7950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:53.871047 7950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:53.871515 7950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:53.871829 7950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:53.872088 7950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:53.872349 7950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:53.907928 7950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:53.909260 7950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:53.910316 7950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:53.928650 7964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:53.928655 7961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:53.930663 7950 server_base.cc:1034] running on GCE node
W20260812 03:05:53.929095 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:55.123461 7950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:55.126785 7950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:55.128237 7950 hybrid_clock.cc:648] HybridClock initialized: now 1786503955128210 us; error 50 us; skew 500 ppm
I20260812 03:05:55.129101 7950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:55 dist-test-slave-zpvx krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:55.135155 7950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:55.140513 7950 webserver.cc:460] Webserver started at http://127.0.0.1:46831/ using document root <none> and password file <none>
I20260812 03:05:55.141453 7950 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:55.141702 7950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:55.142189 7950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:55.147497 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "89b98d4b10cd401bab6673d8cad081b1"
format_stamp: "Formatted at 2026-08-12 03:05:55 on dist-test-slave-zpvx"
server_key: "e4feca63acc3a25a4f10e3533d4adb1d"
server_key_iv: "52db56f236b88341f6a245b0212cb42b"
server_key_version: "encryptionkey@0"
I20260812 03:05:55.148784 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "89b98d4b10cd401bab6673d8cad081b1"
format_stamp: "Formatted at 2026-08-12 03:05:55 on dist-test-slave-zpvx"
server_key: "e4feca63acc3a25a4f10e3533d4adb1d"
server_key_iv: "52db56f236b88341f6a245b0212cb42b"
server_key_version: "encryptionkey@0"
I20260812 03:05:55.156070 7950 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 03:05:55.161742 7984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:55.162786 7950 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:05:55.163123 7950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "89b98d4b10cd401bab6673d8cad081b1"
format_stamp: "Formatted at 2026-08-12 03:05:55 on dist-test-slave-zpvx"
server_key: "e4feca63acc3a25a4f10e3533d4adb1d"
server_key_iv: "52db56f236b88341f6a245b0212cb42b"
server_key_version: "encryptionkey@0"
I20260812 03:05:55.163507 7950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:55.189220 7950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:55.195088 7950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:55.195569 7950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:55.265058 7950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42489
I20260812 03:05:55.265159 8091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42489 every 8 connection(s)
I20260812 03:05:55.267838 7950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:55.273061 8094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:55.276882 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 7950
I20260812 03:05:55.277457 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:55.279031 10713 external_mini_cluster.cc:1428] Setting key ced4e04986e98870653ac9791760f137
I20260812 03:05:55.293953 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:55.281981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:46103 (local address 127.0.0.1:42489)
0812 03:05:55.284847 (+ 2866us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:55.284874 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:55.284928 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:05:55.288933 (+ 4005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:55.288973 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:55.289283 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:55.289599 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:55.291367 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:55.291391 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:55.291420 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:55.291461 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:55.291568 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:55.291626 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:55.291841 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:55.292139 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 03:05:55.292330 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2436,"thread_start_us":1886,"threads_started":1}
I20260812 03:05:55.296283 8094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1: Bootstrap starting.
I20260812 03:05:55.298549 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:55.301894 8094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:55.303591 8094 log.cc:824] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:55.309240 8094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1: No bootstrap required, opened a new log
I20260812 03:05:55.322588 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:55.317148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:60831 (local address 127.0.0.1:42489)
0812 03:05:55.317576 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:55.317591 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:55.318628 (+ 1037us) server_negotiation.cc:407] Connection header received
0812 03:05:55.319003 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:55.319023 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:55.319160 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:55.319326 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:55.320767 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:55.320791 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:55.320806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:05:55.320847 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:55.320932 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:05:55.320983 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:55.321198 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:55.321458 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 03:05:55.321596 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 03:05:55.330585 8094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b98d4b10cd401bab6673d8cad081b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260812 03:05:55.331287 8094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:55.331548 8094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b98d4b10cd401bab6673d8cad081b1, State: Initialized, Role: FOLLOWER
I20260812 03:05:55.332221 8094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b98d4b10cd401bab6673d8cad081b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260812 03:05:55.332926 8094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:55.333238 8094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:55.333606 8094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:55.337698 8094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b98d4b10cd401bab6673d8cad081b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260812 03:05:55.338425 8094 leader_election.cc:302] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b98d4b10cd401bab6673d8cad081b1; no voters:
I20260812 03:05:55.340687 8094 leader_election.cc:288] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:55.341101 8112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:55.343575 8112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [term 1 LEADER]: Becoming Leader. State: Replica: 89b98d4b10cd401bab6673d8cad081b1, State: Running, Role: LEADER
I20260812 03:05:55.344493 8112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b98d4b10cd401bab6673d8cad081b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } }
I20260812 03:05:55.345824 8094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:55.358708 8116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b98d4b10cd401bab6673d8cad081b1. Latest consensus state: current_term: 1 leader_uuid: "89b98d4b10cd401bab6673d8cad081b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b98d4b10cd401bab6673d8cad081b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } } }
I20260812 03:05:55.359508 8116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:55.358771 8115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89b98d4b10cd401bab6673d8cad081b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b98d4b10cd401bab6673d8cad081b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42489 } } }
I20260812 03:05:55.360584 8115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:55.365854 8126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:55.378664 8126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:55.396173 8126 catalog_manager.cc:1347] Generated new cluster ID: 8bc76f3534d8411c927a85a84d30a9c3
I20260812 03:05:55.396451 8126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:55.415095 8126 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:55.415809 8126 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:55.433362 8126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89b98d4b10cd401bab6673d8cad081b1: Generated new TSK 0
I20260812 03:05:55.434502 8126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:55.456713 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 7950
2026-08-12T03:05:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (3576 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:55 dist-test-slave-zpvx krb5kdc[8147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:55 dist-test-slave-zpvx krb5kdc[8147](info): set up 2 sockets
Aug 12 03:05:55 dist-test-slave-zpvx krb5kdc[8147](info): commencing operation
krb5kdc: starting...
W20260812 03:05:56.852186 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:05:57 dist-test-slave-zpvx krb5kdc[8147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:05:57.089269 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:05:57.397984 8170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:05:57.398634 8170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:05:57.399290 8170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:05:57.429782 8170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:05:57.430169 8170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:05:57.430469 8170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:05:57.430720 8170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:05:57.430960 8170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:05:57.466388 8170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:05:57.467747 8170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:05:57.468906 8170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:05:57.485950 8181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:57.485944 8185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:05:57.486470 8180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:05:57.487829 8170 server_base.cc:1034] running on GCE node
I20260812 03:05:58.673449 8170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:05:58.676649 8170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:05:58.678086 8170 hybrid_clock.cc:648] HybridClock initialized: now 1786503958678026 us; error 84 us; skew 500 ppm
I20260812 03:05:58.678869 8170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:05:58 dist-test-slave-zpvx krb5kdc[8147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:05:58.685225 8170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:05:58.691964 8170 webserver.cc:460] Webserver started at http://127.0.0.1:40981/ using document root <none> and password file <none>
I20260812 03:05:58.693315 8170 fs_manager.cc:359] Metadata directory not provided
I20260812 03:05:58.693545 8170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:05:58.694187 8170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:05:58.700050 8170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "ceb959add59243bfaddf3dfe9792d4f6"
format_stamp: "Formatted at 2026-08-12 03:05:58 on dist-test-slave-zpvx"
server_key: "97b5dd53ad9e874c4fb07bc71a8ebb50"
server_key_iv: "d8897df24542ec5339feafd28a934e4f"
server_key_version: "encryptionkey@0"
I20260812 03:05:58.701329 8170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "ceb959add59243bfaddf3dfe9792d4f6"
format_stamp: "Formatted at 2026-08-12 03:05:58 on dist-test-slave-zpvx"
server_key: "97b5dd53ad9e874c4fb07bc71a8ebb50"
server_key_iv: "d8897df24542ec5339feafd28a934e4f"
server_key_version: "encryptionkey@0"
I20260812 03:05:58.708664 8170 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:05:58.714135 8207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:05:58.715178 8170 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:05:58.715492 8170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "ceb959add59243bfaddf3dfe9792d4f6"
format_stamp: "Formatted at 2026-08-12 03:05:58 on dist-test-slave-zpvx"
server_key: "97b5dd53ad9e874c4fb07bc71a8ebb50"
server_key_iv: "d8897df24542ec5339feafd28a934e4f"
server_key_version: "encryptionkey@0"
I20260812 03:05:58.715852 8170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:05:58.750856 8170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:05:58.756228 8170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:05:58.756738 8170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:05:58.824815 8170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39885
I20260812 03:05:58.824909 8304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39885 every 8 connection(s)
I20260812 03:05:58.827543 8170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:05:58.832970 8307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:05:58.835079 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 8170
I20260812 03:05:58.835695 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:05:58.837159 10713 external_mini_cluster.cc:1428] Setting key bd9ff77987b4ad66659a51ed30a4917a
I20260812 03:05:58.861639 8307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6: Bootstrap starting.
Aug 12 03:05:58 dist-test-slave-zpvx krb5kdc[8147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 03:05:58.867245 8307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6: Neither blocks nor log segments found. Creating new log.
I20260812 03:05:58.869025 8307 log.cc:824] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6: Log is configured to *not* fsync() on all Append() calls
I20260812 03:05:58.874835 8307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6: No bootstrap required, opened a new log
I20260812 03:05:58.875334 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:58.839712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:44927 (local address 127.0.0.1:39885)
0812 03:05:58.842504 (+ 2792us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:58.842530 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:58.842583 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:05:58.846503 (+ 3920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:58.846549 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:58.846910 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:58.847243 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:58.850798 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:58.852554 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:58.855372 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:58.856227 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:58.856546 (+ 319us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:58.863559 (+ 7013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:58.863586 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:58.863614 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:58.863730 (+ 116us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:58.867982 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:58.869419 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:58.869442 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:58.869458 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:58.869704 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:58.870713 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:58.870732 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:58.870746 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:58.871438 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:58.872007 (+ 569us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:58.872887 (+ 880us) server_negotiation.cc:299] Negotiation successful
0812 03:05:58.873398 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2354,"thread_start_us":1864,"threads_started":1}
I20260812 03:05:58.880170 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:05:58.896472 8307 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb959add59243bfaddf3dfe9792d4f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260812 03:05:58.897192 8307 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:05:58.897440 8307 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb959add59243bfaddf3dfe9792d4f6, State: Initialized, Role: FOLLOWER
I20260812 03:05:58.898132 8307 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb959add59243bfaddf3dfe9792d4f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260812 03:05:58.898857 8307 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:05:58.899184 8307 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:05:58.899559 8307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:05:58.904706 8307 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb959add59243bfaddf3dfe9792d4f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260812 03:05:58.905435 8307 leader_election.cc:302] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceb959add59243bfaddf3dfe9792d4f6; no voters:
I20260812 03:05:58.907313 8307 leader_election.cc:288] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:05:58.907688 8331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:05:58.910490 8331 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [term 1 LEADER]: Becoming Leader. State: Replica: ceb959add59243bfaddf3dfe9792d4f6, State: Running, Role: LEADER
I20260812 03:05:58.911249 8331 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb959add59243bfaddf3dfe9792d4f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260812 03:05:58.912405 8307 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:05:58.925546 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:05:58.893549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:39069 (local address 127.0.0.1:39885)
0812 03:05:58.894107 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0812 03:05:58.894137 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:05:58.895045 (+ 908us) server_negotiation.cc:407] Connection header received
0812 03:05:58.895357 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:05:58.895392 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:05:58.895612 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:05:58.895793 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:05:58.898254 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:58.899793 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:58.901607 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:05:58.902527 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:05:58.902808 (+ 281us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:05:58.906617 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:05:58.906656 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:05:58.906694 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:05:58.906855 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:05:58.912953 (+ 6098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:58.915114 (+ 2161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:58.915158 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:58.915192 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:58.915533 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:05:58.918010 (+ 2477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:05:58.918126 (+ 116us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:05:58.918159 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:05:58.919320 (+ 1161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:05:58.920022 (+ 702us) server_negotiation.cc:1035] Waiting for connection context
0812 03:05:58.922646 (+ 2624us) server_negotiation.cc:299] Negotiation successful
0812 03:05:58.923104 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260812 03:05:58.927541 8335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb959add59243bfaddf3dfe9792d4f6. Latest consensus state: current_term: 1 leader_uuid: "ceb959add59243bfaddf3dfe9792d4f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb959add59243bfaddf3dfe9792d4f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } } }
I20260812 03:05:58.928227 8335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:58.924494 8332 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceb959add59243bfaddf3dfe9792d4f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceb959add59243bfaddf3dfe9792d4f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } } }
I20260812 03:05:58.930576 8332 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6 [sys.catalog]: This master's current role is: LEADER
I20260812 03:05:58.935304 8348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:05:58.947619 8348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:05:58.964352 8348 catalog_manager.cc:1347] Generated new cluster ID: 5d45ac20f1624c8abe1f0caac823314e
I20260812 03:05:58.964689 8348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:05:58.984360 8348 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:05:58.986843 8348 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:05:59.005714 8348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceb959add59243bfaddf3dfe9792d4f6: Generated new TSK 0
I20260812 03:05:59.006701 8348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:05:59.022544 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 8170
W20260812 03:05:59.068001 8143 connection.cc:535] client connection to 127.0.0.1:39885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39885): Connection reset by peer (error 104)
2026-08-12T03:05:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (3577 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:05:59 dist-test-slave-zpvx krb5kdc[8376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:05:59 dist-test-slave-zpvx krb5kdc[8376](info): set up 2 sockets
Aug 12 03:05:59 dist-test-slave-zpvx krb5kdc[8376](info): commencing operation
krb5kdc: starting...
W20260812 03:06:00.423520 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:00 dist-test-slave-zpvx krb5kdc[8376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:06:00.688683 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:01.006163 8399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:01.006753 8399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:01.007341 8399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:01.038810 8399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:01.039155 8399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:01.039399 8399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:01.039592 8399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:01.039812 8399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:01.076104 8399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:01.077356 8399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:01.078436 8399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:01.096434 8415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:01.096746 8410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:01.098206 8399 server_base.cc:1034] running on GCE node
W20260812 03:06:01.097657 8420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:02.284986 8399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:02.288265 8399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:02.289723 8399 hybrid_clock.cc:648] HybridClock initialized: now 1786503962289662 us; error 80 us; skew 500 ppm
I20260812 03:06:02.290518 8399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:02 dist-test-slave-zpvx krb5kdc[8376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:02.303784 8399 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:06:02.309253 8399 webserver.cc:460] Webserver started at http://127.0.0.1:40607/ using document root <none> and password file <none>
I20260812 03:06:02.310164 8399 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:02.310351 8399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:02.310767 8399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:02.316103 8399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "3f4159c57df540de92e1f120a62d5037"
format_stamp: "Formatted at 2026-08-12 03:06:02 on dist-test-slave-zpvx"
server_key: "e0a974254dcc017d2c9adaa275ca1d1b"
server_key_iv: "1f8657eb2c12bc0f58918f194cb0c863"
server_key_version: "encryptionkey@0"
I20260812 03:06:02.317344 8399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "3f4159c57df540de92e1f120a62d5037"
format_stamp: "Formatted at 2026-08-12 03:06:02 on dist-test-slave-zpvx"
server_key: "e0a974254dcc017d2c9adaa275ca1d1b"
server_key_iv: "1f8657eb2c12bc0f58918f194cb0c863"
server_key_version: "encryptionkey@0"
I20260812 03:06:02.324754 8399 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 03:06:02.330528 8441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:02.331619 8399 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260812 03:06:02.331961 8399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "3f4159c57df540de92e1f120a62d5037"
format_stamp: "Formatted at 2026-08-12 03:06:02 on dist-test-slave-zpvx"
server_key: "e0a974254dcc017d2c9adaa275ca1d1b"
server_key_iv: "1f8657eb2c12bc0f58918f194cb0c863"
server_key_version: "encryptionkey@0"
I20260812 03:06:02.332351 8399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:02.384042 8399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:02.390026 8399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:02.390515 8399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:02.460868 8399 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42009
I20260812 03:06:02.460965 8531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42009 every 8 connection(s)
I20260812 03:06:02.463673 8399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:02.469116 8533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:02.472126 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 8399
I20260812 03:06:02.472661 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:02.473743 10713 external_mini_cluster.cc:1428] Setting key ca835e0f67e62b5706b0f0885fe03731
I20260812 03:06:02.496920 8533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037: Bootstrap starting.
I20260812 03:06:02.498916 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:02.475943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:37729 (local address 127.0.0.1:42009)
0812 03:06:02.478605 (+ 2662us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:02.478645 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:02.478750 (+ 105us) server_negotiation.cc:407] Connection header received
0812 03:06:02.483099 (+ 4349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:02.483157 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:02.483699 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:02.484163 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:02.485251 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:02.487767 (+ 2516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:02.490203 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:02.491610 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:02.491959 (+ 349us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:02.494088 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:02.494123 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:02.494169 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:02.495278 (+ 1109us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:02.495457 (+ 179us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:02.495550 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:02.495817 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:02.496229 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 03:06:02.496455 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2093,"thread_start_us":1652,"threads_started":1}
I20260812 03:06:02.503741 8533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:02.503861 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:02.505589 8533 log.cc:824] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:02.511088 8533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037: No bootstrap required, opened a new log
I20260812 03:06:02.529436 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:02.518255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:55187 (local address 127.0.0.1:42009)
0812 03:06:02.518664 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:02.518682 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:02.519728 (+ 1046us) server_negotiation.cc:407] Connection header received
0812 03:06:02.520133 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:02.520153 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:02.520328 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:02.520554 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:02.521646 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:02.522636 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:02.524547 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:02.525134 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:02.525425 (+ 291us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:02.527223 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:02.527244 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:02.527271 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:02.527317 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:02.527397 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:02.527449 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:02.527664 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:02.528081 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 03:06:02.528203 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 03:06:02.532967 8533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4159c57df540de92e1f120a62d5037" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42009 } }
I20260812 03:06:02.533718 8533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:02.534019 8533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4159c57df540de92e1f120a62d5037, State: Initialized, Role: FOLLOWER
I20260812 03:06:02.534933 8533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4159c57df540de92e1f120a62d5037" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42009 } }
I20260812 03:06:02.535557 8533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:02.535854 8533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:02.536166 8533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:02.540494 8533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4159c57df540de92e1f120a62d5037" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42009 } }
I20260812 03:06:02.541329 8533 leader_election.cc:302] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f4159c57df540de92e1f120a62d5037; no voters:
I20260812 03:06:02.543711 8533 leader_election.cc:288] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:02.544144 8545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:02.546657 8545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [term 1 LEADER]: Becoming Leader. State: Replica: 3f4159c57df540de92e1f120a62d5037, State: Running, Role: LEADER
I20260812 03:06:02.547665 8545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4159c57df540de92e1f120a62d5037" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42009 } }
I20260812 03:06:02.548945 8533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:02.559420 8547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f4159c57df540de92e1f120a62d5037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4159c57df540de92e1f120a62d5037" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42009 } } }
I20260812 03:06:02.559433 8548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f4159c57df540de92e1f120a62d5037. Latest consensus state: current_term: 1 leader_uuid: "3f4159c57df540de92e1f120a62d5037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f4159c57df540de92e1f120a62d5037" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42009 } } }
I20260812 03:06:02.560581 8547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:02.560603 8548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:02.569288 8565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:02.583276 8565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:02.600880 8565 catalog_manager.cc:1347] Generated new cluster ID: e39cdb51ef464d65b573f3a8e8623c51
I20260812 03:06:02.601224 8565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:02.619611 8565 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:02.621904 8565 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:02.645339 8565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f4159c57df540de92e1f120a62d5037: Generated new TSK 0
I20260812 03:06:02.647289 8565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:02.657701 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 8399
W20260812 03:06:02.697774 8367 connection.cc:535] client connection to 127.0.0.1:42009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42009): Connection reset by peer (error 104)
2026-08-12T03:06:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (3629 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:02 dist-test-slave-zpvx krb5kdc[8586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:02 dist-test-slave-zpvx krb5kdc[8586](info): set up 2 sockets
Aug 12 03:06:02 dist-test-slave-zpvx krb5kdc[8586](info): commencing operation
krb5kdc: starting...
W20260812 03:06:04.057516 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.011s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:04 dist-test-slave-zpvx krb5kdc[8586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 03:06:04.301349 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:04.612347 8609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:04.612932 8609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:04.613521 8609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:04.644244 8609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:04.644589 8609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:04.644824 8609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:04.645016 8609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:04.645241 8609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:04.681452 8609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:04.682576 8609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:04.683640 8609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:04.705809 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:04.706094 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:04.707417 8609 server_base.cc:1034] running on GCE node
W20260812 03:06:04.707341 8630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:05.894691 8609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:05.897768 8609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:05.899250 8609 hybrid_clock.cc:648] HybridClock initialized: now 1786503965899213 us; error 74 us; skew 500 ppm
I20260812 03:06:05.900306 8609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:05 dist-test-slave-zpvx krb5kdc[8586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:05.908763 8609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 03:06:05.918102 8609 webserver.cc:460] Webserver started at http://127.0.0.1:43973/ using document root <none> and password file <none>
I20260812 03:06:05.919422 8609 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:05.919735 8609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:05.920470 8609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:05.929093 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "ea40e3ba4a584e3f81c1a13adc3cf146"
format_stamp: "Formatted at 2026-08-12 03:06:05 on dist-test-slave-zpvx"
server_key: "fd06a35bd17964b2294732c4beb45553"
server_key_iv: "8ae4d2e9bf04b5d640da1ad3252f329f"
server_key_version: "encryptionkey@0"
I20260812 03:06:05.931015 8609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "ea40e3ba4a584e3f81c1a13adc3cf146"
format_stamp: "Formatted at 2026-08-12 03:06:05 on dist-test-slave-zpvx"
server_key: "fd06a35bd17964b2294732c4beb45553"
server_key_iv: "8ae4d2e9bf04b5d640da1ad3252f329f"
server_key_version: "encryptionkey@0"
I20260812 03:06:05.943993 8609 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.000s
I20260812 03:06:05.950667 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:05.951769 8609 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 03:06:05.952121 8609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "ea40e3ba4a584e3f81c1a13adc3cf146"
format_stamp: "Formatted at 2026-08-12 03:06:05 on dist-test-slave-zpvx"
server_key: "fd06a35bd17964b2294732c4beb45553"
server_key_iv: "8ae4d2e9bf04b5d640da1ad3252f329f"
server_key_version: "encryptionkey@0"
I20260812 03:06:05.952575 8609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:05.979759 8609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:05.985667 8609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:05.986140 8609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:06.062844 8609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36193
I20260812 03:06:06.062960 8748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36193 every 8 connection(s)
I20260812 03:06:06.065604 8609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:06.071061 8750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:06.071591 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 8609
I20260812 03:06:06.072180 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:06.073964 10713 external_mini_cluster.cc:1428] Setting key d72c8971fb534e98036d18ee949e7f79
I20260812 03:06:06.089509 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:06.076678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:33733 (local address 127.0.0.1:36193)
0812 03:06:06.079476 (+ 2798us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:06.079503 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:06.079557 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:06:06.084170 (+ 4613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:06.084214 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:06.084587 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:06.084938 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:06.086791 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:06.086814 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:06.086843 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:06.086884 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:06.086993 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:06.087050 (+ 57us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:06.087274 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:06.087576 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 03:06:06.087760 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2361,"thread_start_us":1848,"threads_started":1}
I20260812 03:06:06.094302 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:06.099112 8750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146: Bootstrap starting.
I20260812 03:06:06.106730 8750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:06.109098 8750 log.cc:824] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:06.115939 8750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146: No bootstrap required, opened a new log
I20260812 03:06:06.123180 8758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:06.116724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.244:36253 (local address 127.0.0.1:36193)
0812 03:06:06.117289 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:06.117312 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:06.118505 (+ 1193us) server_negotiation.cc:407] Connection header received
0812 03:06:06.118790 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:06.118818 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:06.119001 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:06.119198 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:06.120673 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:06.120703 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:06.120733 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:06.120794 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:06.120906 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:06.120982 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:06.121214 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:06.121601 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 03:06:06.121738 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 03:06:06.138293 8750 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260812 03:06:06.139019 8750 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:06.139321 8750 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea40e3ba4a584e3f81c1a13adc3cf146, State: Initialized, Role: FOLLOWER
I20260812 03:06:06.140151 8750 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260812 03:06:06.140710 8750 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:06.140998 8750 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:06.141304 8750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:06.145316 8750 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260812 03:06:06.145987 8750 leader_election.cc:302] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea40e3ba4a584e3f81c1a13adc3cf146; no voters:
I20260812 03:06:06.147953 8750 leader_election.cc:288] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:06.148460 8775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:06.151166 8775 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [term 1 LEADER]: Becoming Leader. State: Replica: ea40e3ba4a584e3f81c1a13adc3cf146, State: Running, Role: LEADER
I20260812 03:06:06.151935 8775 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } }
I20260812 03:06:06.152885 8750 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:06.163779 8777 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea40e3ba4a584e3f81c1a13adc3cf146. Latest consensus state: current_term: 1 leader_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } } }
I20260812 03:06:06.163897 8776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea40e3ba4a584e3f81c1a13adc3cf146" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36193 } } }
I20260812 03:06:06.164870 8776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:06.166347 8777 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:06.169955 8795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:06.179936 8795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:06.197031 8795 catalog_manager.cc:1347] Generated new cluster ID: 1494190e7a79458d8648f1132e10e3a9
I20260812 03:06:06.197362 8795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:06.224234 8795 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:06.225003 8795 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:06.244943 8795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea40e3ba4a584e3f81c1a13adc3cf146: Generated new TSK 0
I20260812 03:06:06.246105 8795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:06.255110 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 8609
2026-08-12T03:06:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (3595 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:06 dist-test-slave-zpvx krb5kdc[8813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:06 dist-test-slave-zpvx krb5kdc[8813](info): set up 2 sockets
Aug 12 03:06:06 dist-test-slave-zpvx krb5kdc[8813](info): commencing operation
krb5kdc: starting...
W20260812 03:06:07.644865 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:07 dist-test-slave-zpvx krb5kdc[8813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:07.893258 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36613
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:43485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:08.202298 8837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:08.202975 8837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:08.203651 8837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:08.234174 8837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:08.234579 8837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:08.234906 8837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:08.235169 8837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:08.235430 8837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:08.271243 8837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:43485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:36613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:08.272496 8837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:08.273535 8837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:08.292609 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:08.292804 8847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:08.293710 8853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:09.473591 8851 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:06:09.473696 8837 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:06:09.477437 8837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:09.480657 8837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:09.482098 8837 hybrid_clock.cc:648] HybridClock initialized: now 1786503969482052 us; error 69 us; skew 500 ppm
I20260812 03:06:09.482940 8837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:09 dist-test-slave-zpvx krb5kdc[8813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503969, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:09.493566 8837 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:09.499492 8837 webserver.cc:460] Webserver started at http://127.10.118.126:44385/ using document root <none> and password file <none>
I20260812 03:06:09.500576 8837 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:09.500830 8837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:09.501392 8837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:09.507273 8837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "073e9749cb1a41f8a3399e71fea84b8c"
format_stamp: "Formatted at 2026-08-12 03:06:09 on dist-test-slave-zpvx"
server_key: "2d149800bb2c0c10f370af001c013658"
server_key_iv: "061ed2f9b11d5336d104fec2f1fc0444"
server_key_version: "encryptionkey@0"
I20260812 03:06:09.508625 8837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "073e9749cb1a41f8a3399e71fea84b8c"
format_stamp: "Formatted at 2026-08-12 03:06:09 on dist-test-slave-zpvx"
server_key: "2d149800bb2c0c10f370af001c013658"
server_key_iv: "061ed2f9b11d5336d104fec2f1fc0444"
server_key_version: "encryptionkey@0"
I20260812 03:06:09.516532 8837 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 03:06:09.522647 8866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:09.523806 8837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 03:06:09.524226 8837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "073e9749cb1a41f8a3399e71fea84b8c"
format_stamp: "Formatted at 2026-08-12 03:06:09 on dist-test-slave-zpvx"
server_key: "2d149800bb2c0c10f370af001c013658"
server_key_iv: "061ed2f9b11d5336d104fec2f1fc0444"
server_key_version: "encryptionkey@0"
I20260812 03:06:09.524693 8837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:09.561094 8837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:09.567049 8837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:09.567580 8837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:09.642416 8837 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:36613
I20260812 03:06:09.642535 8960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:36613 every 8 connection(s)
I20260812 03:06:09.645491 8837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:09.649672 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 8837
I20260812 03:06:09.650254 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:09.651736 10713 external_mini_cluster.cc:1428] Setting key 073eb22a9106263ad95a852a362b1c72
I20260812 03:06:09.651799 8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 03:06:09 dist-test-slave-zpvx krb5kdc[8813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:09.679862 8961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c: Bootstrap starting.
I20260812 03:06:09.685148 8961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:09.686815 8961 log.cc:824] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:09.692062 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:09.654424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41333 (local address 127.10.118.126:36613)
0812 03:06:09.657483 (+ 3059us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:09.657512 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:09.657566 (+ 54us) server_negotiation.cc:407] Connection header received
0812 03:06:09.662465 (+ 4899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:09.662507 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:09.662916 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:09.663275 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:09.667093 (+ 3818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:09.669024 (+ 1931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:09.671917 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:09.672849 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:09.673073 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:09.680190 (+ 7117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:09.680216 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:09.680248 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:09.680411 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:09.684728 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:09.686186 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:09.686208 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:09.686226 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:09.686479 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:09.687442 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:09.687461 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:09.687474 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:09.688162 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:09.688759 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:09.689661 (+ 902us) server_negotiation.cc:299] Negotiation successful
0812 03:06:09.690189 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2593,"thread_start_us":2070,"threads_started":1}
I20260812 03:06:09.692708 8961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c: No bootstrap required, opened a new log
I20260812 03:06:09.697157 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:09.714370 8961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e9749cb1a41f8a3399e71fea84b8c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36613 } }
I20260812 03:06:09.715026 8961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:09.715333 8961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 073e9749cb1a41f8a3399e71fea84b8c, State: Initialized, Role: FOLLOWER
I20260812 03:06:09.716074 8961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e9749cb1a41f8a3399e71fea84b8c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36613 } }
I20260812 03:06:09.716670 8961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:09.716954 8961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:09.717288 8961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:09.721357 8961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e9749cb1a41f8a3399e71fea84b8c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36613 } }
I20260812 03:06:09.722066 8961 leader_election.cc:302] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 073e9749cb1a41f8a3399e71fea84b8c; no voters:
I20260812 03:06:09.724320 8961 leader_election.cc:288] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:09.724750 8976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:09.727317 8976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [term 1 LEADER]: Becoming Leader. State: Replica: 073e9749cb1a41f8a3399e71fea84b8c, State: Running, Role: LEADER
I20260812 03:06:09.728181 8976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e9749cb1a41f8a3399e71fea84b8c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36613 } }
I20260812 03:06:09.730540 8961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:09.737823 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:09.709889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47597 (local address 127.10.118.126:36613)
0812 03:06:09.710267 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:09.710285 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:09.711384 (+ 1099us) server_negotiation.cc:407] Connection header received
0812 03:06:09.711780 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:09.711807 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:09.712007 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:09.712182 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:09.714486 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:09.715633 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:09.717731 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:09.718323 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:09.718608 (+ 285us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:09.722272 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:09.722299 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:09.722316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:09.722441 (+ 125us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:09.727244 (+ 4803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:09.729229 (+ 1985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:09.729271 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:09.729300 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:09.729546 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:09.730555 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:09.730585 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:09.730610 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:09.731328 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:09.734211 (+ 2883us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:09.734735 (+ 524us) server_negotiation.cc:299] Negotiation successful
0812 03:06:09.735226 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 03:06:09.739748 8978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 073e9749cb1a41f8a3399e71fea84b8c. Latest consensus state: current_term: 1 leader_uuid: "073e9749cb1a41f8a3399e71fea84b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e9749cb1a41f8a3399e71fea84b8c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36613 } } }
I20260812 03:06:09.740845 8978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:09.740768 8979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "073e9749cb1a41f8a3399e71fea84b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "073e9749cb1a41f8a3399e71fea84b8c" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36613 } } }
I20260812 03:06:09.742175 8979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:09.749791 8993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:09.760555 8993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:09.778544 8993 catalog_manager.cc:1347] Generated new cluster ID: dd2dc16429cf480998e7049ff527e455
I20260812 03:06:09.778834 8993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:09.794730 8993 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:09.797276 8993 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:09.818780 8993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 073e9749cb1a41f8a3399e71fea84b8c: Generated new TSK 0
I20260812 03:06:09.820025 8993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:09.836246 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 8837
W20260812 03:06:09.876091 8810 connection.cc:535] client connection to 127.10.118.126:36613 recv error: Network error: failed to read from TLS socket (remote: 127.10.118.126:36613): Connection reset by peer (error 104)
2026-08-12T03:06:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (3582 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:09 dist-test-slave-zpvx krb5kdc[9018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:09 dist-test-slave-zpvx krb5kdc[9018](info): set up 2 sockets
Aug 12 03:06:09 dist-test-slave-zpvx krb5kdc[9018](info): commencing operation
krb5kdc: starting...
W20260812 03:06:11.235255 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:11 dist-test-slave-zpvx krb5kdc[9018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:11.476492 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36929
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:36929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:11.787604 9043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:11.788275 9043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:11.788936 9043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:11.819625 9043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:11.819944 9043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:11.820181 9043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:11.820444 9043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:11.820648 9043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:11.856182 9043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:36929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:11.857343 9043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:11.858353 9043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:11.876154 9063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:11.876199 9060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:11.876214 9053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:11.877828 9043 server_base.cc:1034] running on GCE node
I20260812 03:06:13.067729 9043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:13.070456 9043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:13.071807 9043 hybrid_clock.cc:648] HybridClock initialized: now 1786503973071785 us; error 44 us; skew 500 ppm
I20260812 03:06:13.072700 9043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:13 dist-test-slave-zpvx krb5kdc[9018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503973, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:13.078702 9043 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:13.083845 9043 webserver.cc:460] Webserver started at http://127.10.118.126:38411/ using document root <none> and password file <none>
I20260812 03:06:13.084868 9043 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:13.085107 9043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:13.085587 9043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:13.090965 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "ba905888f68e4370ba463ce204fba0d4"
format_stamp: "Formatted at 2026-08-12 03:06:13 on dist-test-slave-zpvx"
server_key: "ad2f6b38d3b0cecf8fb48bfc5eaf31d1"
server_key_iv: "fd19b149ce51a1b9a38ce0542bf49770"
server_key_version: "encryptionkey@0"
I20260812 03:06:13.092181 9043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "ba905888f68e4370ba463ce204fba0d4"
format_stamp: "Formatted at 2026-08-12 03:06:13 on dist-test-slave-zpvx"
server_key: "ad2f6b38d3b0cecf8fb48bfc5eaf31d1"
server_key_iv: "fd19b149ce51a1b9a38ce0542bf49770"
server_key_version: "encryptionkey@0"
I20260812 03:06:13.099740 9043 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 03:06:13.106155 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:13.107371 9043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20260812 03:06:13.107664 9043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "ba905888f68e4370ba463ce204fba0d4"
format_stamp: "Formatted at 2026-08-12 03:06:13 on dist-test-slave-zpvx"
server_key: "ad2f6b38d3b0cecf8fb48bfc5eaf31d1"
server_key_iv: "fd19b149ce51a1b9a38ce0542bf49770"
server_key_version: "encryptionkey@0"
I20260812 03:06:13.108000 9043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:13.146917 9043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:13.153295 9043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:13.153792 9043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:13.225360 9043 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:36929
I20260812 03:06:13.225450 9167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:36929 every 8 connection(s)
I20260812 03:06:13.228291 9043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:13.233713 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:13.237625 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 9043
I20260812 03:06:13.238142 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:13.239225 10713 external_mini_cluster.cc:1428] Setting key 87054112f99ae4e5a59ea1d674851bfb
I20260812 03:06:13.256603 9169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4: Bootstrap starting.
I20260812 03:06:13.262516 9169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:13.262871 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:13.241711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56467 (local address 127.10.118.126:36929)
0812 03:06:13.244616 (+ 2905us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:13.244645 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:13.244702 (+ 57us) server_negotiation.cc:407] Connection header received
0812 03:06:13.248688 (+ 3986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:13.248730 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:13.249101 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:13.249441 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:13.250960 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:13.252691 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:13.255827 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:13.256802 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:13.257148 (+ 346us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:13.259174 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:13.259198 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:13.259233 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:13.259993 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:13.260110 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:13.260169 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:13.260448 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:13.260949 (+ 501us) server_negotiation.cc:299] Negotiation successful
0812 03:06:13.261168 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2373,"thread_start_us":1877,"threads_started":1}
I20260812 03:06:13.265077 9169 log.cc:824] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:13.268146 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:13.270197 9169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4: No bootstrap required, opened a new log
I20260812 03:06:13.292127 9169 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba905888f68e4370ba463ce204fba0d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36929 } }
I20260812 03:06:13.293129 9169 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:13.293484 9169 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba905888f68e4370ba463ce204fba0d4, State: Initialized, Role: FOLLOWER
I20260812 03:06:13.294360 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:13.281614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45085 (local address 127.10.118.126:36929)
0812 03:06:13.282143 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:13.282170 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:13.283265 (+ 1095us) server_negotiation.cc:407] Connection header received
0812 03:06:13.283558 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:13.283586 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:13.283816 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:13.284006 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:13.285011 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:13.286542 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:13.288447 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:13.289512 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:13.289803 (+ 291us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:13.291422 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:13.291454 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:13.291478 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:13.291545 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:13.291656 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:13.291727 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:13.291952 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:13.292435 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 03:06:13.292559 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 03:06:13.294440 9169 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba905888f68e4370ba463ce204fba0d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36929 } }
I20260812 03:06:13.295166 9169 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:13.295523 9169 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:13.295879 9169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:13.300100 9169 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba905888f68e4370ba463ce204fba0d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36929 } }
I20260812 03:06:13.300944 9169 leader_election.cc:302] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba905888f68e4370ba463ce204fba0d4; no voters:
I20260812 03:06:13.302990 9169 leader_election.cc:288] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:13.303448 9183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:13.305662 9183 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [term 1 LEADER]: Becoming Leader. State: Replica: ba905888f68e4370ba463ce204fba0d4, State: Running, Role: LEADER
I20260812 03:06:13.306556 9183 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba905888f68e4370ba463ce204fba0d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36929 } }
I20260812 03:06:13.307505 9169 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:13.319006 9184 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba905888f68e4370ba463ce204fba0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba905888f68e4370ba463ce204fba0d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36929 } } }
I20260812 03:06:13.319911 9184 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:13.321455 9185 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba905888f68e4370ba463ce204fba0d4. Latest consensus state: current_term: 1 leader_uuid: "ba905888f68e4370ba463ce204fba0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba905888f68e4370ba463ce204fba0d4" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36929 } } }
I20260812 03:06:13.322235 9185 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:13.325240 9202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:13.336058 9202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:13.353441 9202 catalog_manager.cc:1347] Generated new cluster ID: f5d77e37059d4663b237931c909da1ba
I20260812 03:06:13.353770 9202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:13.366911 9202 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:13.369343 9202 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:13.385201 9202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba905888f68e4370ba463ce204fba0d4: Generated new TSK 0
I20260812 03:06:13.386124 9202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:13.404934 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 9043
W20260812 03:06:13.443640 9015 connection.cc:535] client connection to 127.10.118.126:36929 recv error: Network error: failed to read from TLS socket (remote: 127.10.118.126:36929): Connection reset by peer (error 104)
2026-08-12T03:06:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (3567 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:13 dist-test-slave-zpvx krb5kdc[9220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:13 dist-test-slave-zpvx krb5kdc[9220](info): set up 2 sockets
Aug 12 03:06:13 dist-test-slave-zpvx krb5kdc[9220](info): commencing operation
krb5kdc: starting...
W20260812 03:06:14.800856 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.006s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:14 dist-test-slave-zpvx krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:15.043215 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38045
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:15.352535 9245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:15.353174 9245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:15.353833 9245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:15.384884 9245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:15.385311 9245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:15.385617 9245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:15.385878 9245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:15.386122 9245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:15.423506 9245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:38045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:38045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:15.424790 9245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:15.425876 9245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:15.446290 9264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:15.446302 9261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:15.446842 9260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:15.448812 9245 server_base.cc:1034] running on GCE node
I20260812 03:06:16.645773 9245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:16.648622 9245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:16.650036 9245 hybrid_clock.cc:648] HybridClock initialized: now 1786503976649991 us; error 58 us; skew 500 ppm
I20260812 03:06:16.650928 9245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:16 dist-test-slave-zpvx krb5kdc[9220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503976, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:16.664206 9245 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:16.669654 9245 webserver.cc:460] Webserver started at http://127.10.118.126:44399/ using document root <none> and password file <none>
I20260812 03:06:16.670645 9245 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:16.670841 9245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:16.671262 9245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:16.676717 9245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "326f443e29ff4316852af9f572417a39"
format_stamp: "Formatted at 2026-08-12 03:06:16 on dist-test-slave-zpvx"
server_key: "45d7dc4ea222f6c69f8a4689b064d316"
server_key_iv: "dede181b7760cc824f46bdf01d651ed2"
server_key_version: "encryptionkey@0"
I20260812 03:06:16.677845 9245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "326f443e29ff4316852af9f572417a39"
format_stamp: "Formatted at 2026-08-12 03:06:16 on dist-test-slave-zpvx"
server_key: "45d7dc4ea222f6c69f8a4689b064d316"
server_key_iv: "dede181b7760cc824f46bdf01d651ed2"
server_key_version: "encryptionkey@0"
I20260812 03:06:16.685662 9245 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:06:16.696401 9279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:16.697990 9245 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.001s
I20260812 03:06:16.698381 9245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "326f443e29ff4316852af9f572417a39"
format_stamp: "Formatted at 2026-08-12 03:06:16 on dist-test-slave-zpvx"
server_key: "45d7dc4ea222f6c69f8a4689b064d316"
server_key_iv: "dede181b7760cc824f46bdf01d651ed2"
server_key_version: "encryptionkey@0"
I20260812 03:06:16.698892 9245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:16.724731 9245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:16.730454 9245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:16.730887 9245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:16.801761 9245 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:38045
I20260812 03:06:16.801864 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:38045 every 8 connection(s)
I20260812 03:06:16.804611 9245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:16.809772 9367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:16.813735 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 9245
I20260812 03:06:16.814226 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:16.815375 10713 external_mini_cluster.cc:1428] Setting key 6ffdf6648808dcecb5a06ca39a4ef93c
I20260812 03:06:16.832464 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:16.817791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37653 (local address 127.10.118.126:38045)
0812 03:06:16.820750 (+ 2959us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:16.820792 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:16.820865 (+ 73us) server_negotiation.cc:407] Connection header received
0812 03:06:16.826575 (+ 5710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:16.826638 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:16.827049 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:16.827488 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:16.829145 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:16.829180 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:16.829221 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:16.829280 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:16.829430 (+ 150us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:16.829512 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:16.829807 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:16.830036 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 03:06:16.830259 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2369,"thread_start_us":1951,"threads_started":1}
I20260812 03:06:16.835246 9367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39: Bootstrap starting.
I20260812 03:06:16.837308 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:16.842576 9367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:16.844976 9367 log.cc:824] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:16.851307 9367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39: No bootstrap required, opened a new log
I20260812 03:06:16.864270 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:16.858477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39411 (local address 127.10.118.126:38045)
0812 03:06:16.859095 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:16.859133 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:16.860213 (+ 1080us) server_negotiation.cc:407] Connection header received
0812 03:06:16.860628 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:16.860646 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:16.860783 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:16.860914 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:16.862441 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:16.862461 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:16.862480 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:16.862523 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:16.862621 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:16.862673 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:16.862843 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:16.863123 (+ 280us) server_negotiation.cc:299] Negotiation successful
0812 03:06:16.863222 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 03:06:16.875464 9367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f443e29ff4316852af9f572417a39" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38045 } }
I20260812 03:06:16.876262 9367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:16.876595 9367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326f443e29ff4316852af9f572417a39, State: Initialized, Role: FOLLOWER
I20260812 03:06:16.877377 9367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f443e29ff4316852af9f572417a39" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38045 } }
I20260812 03:06:16.878088 9367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:16.878398 9367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:16.878786 9367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:16.883608 9367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f443e29ff4316852af9f572417a39" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38045 } }
I20260812 03:06:16.884585 9367 leader_election.cc:302] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 326f443e29ff4316852af9f572417a39; no voters:
I20260812 03:06:16.886351 9367 leader_election.cc:288] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:16.886745 9391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:16.889400 9391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [term 1 LEADER]: Becoming Leader. State: Replica: 326f443e29ff4316852af9f572417a39, State: Running, Role: LEADER
I20260812 03:06:16.890151 9391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f443e29ff4316852af9f572417a39" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38045 } }
I20260812 03:06:16.891233 9367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:16.904836 9394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 326f443e29ff4316852af9f572417a39. Latest consensus state: current_term: 1 leader_uuid: "326f443e29ff4316852af9f572417a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f443e29ff4316852af9f572417a39" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38045 } } }
I20260812 03:06:16.905046 9392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "326f443e29ff4316852af9f572417a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f443e29ff4316852af9f572417a39" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 38045 } } }
I20260812 03:06:16.905691 9394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:16.905810 9392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:16.908820 9410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:16.920178 9410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:16.936838 9410 catalog_manager.cc:1347] Generated new cluster ID: 9c037b0f71c24312b88018845d4fab29
I20260812 03:06:16.937152 9410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:16.952936 9410 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:16.953471 9410 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:16.965541 9410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 326f443e29ff4316852af9f572417a39: Generated new TSK 0
I20260812 03:06:16.966390 9410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:16.980697 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 9245
2026-08-12T03:06:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (3577 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:17 dist-test-slave-zpvx krb5kdc[9435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:17 dist-test-slave-zpvx krb5kdc[9435](info): set up 2 sockets
Aug 12 03:06:17 dist-test-slave-zpvx krb5kdc[9435](info): commencing operation
krb5kdc: starting...
W20260812 03:06:18.402874 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.355s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:18 dist-test-slave-zpvx krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:18.649251 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:39441
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:39441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:18.967808 9460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:18.968449 9460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:18.969135 9460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:18.999275 9460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:18.999670 9460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:18.999965 9460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:19.000219 9460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:19.000494 9460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:19.036521 9460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:39441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:39441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:19.037773 9460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:19.038841 9460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:19.055962 9469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:19.058113 9472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:19.056021 9474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:19.058656 9460 server_base.cc:1034] running on GCE node
I20260812 03:06:20.249082 9460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:20.252944 9460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:20.254555 9460 hybrid_clock.cc:648] HybridClock initialized: now 1786503980254511 us; error 42 us; skew 500 ppm
I20260812 03:06:20.255849 9460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:20 dist-test-slave-zpvx krb5kdc[9435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503980, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:20.263988 9460 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:20.269793 9460 webserver.cc:460] Webserver started at http://127.10.118.126:36001/ using document root <none> and password file <none>
I20260812 03:06:20.270780 9460 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:20.271090 9460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:20.271632 9460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:20.277292 9460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "7e3ad42912744246bff592afbd751261"
format_stamp: "Formatted at 2026-08-12 03:06:20 on dist-test-slave-zpvx"
server_key: "7d2e88f59738026c765e654874e5586d"
server_key_iv: "beb8b835e7837320c02c293c3ab26149"
server_key_version: "encryptionkey@0"
I20260812 03:06:20.278486 9460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "7e3ad42912744246bff592afbd751261"
format_stamp: "Formatted at 2026-08-12 03:06:20 on dist-test-slave-zpvx"
server_key: "7d2e88f59738026c765e654874e5586d"
server_key_iv: "beb8b835e7837320c02c293c3ab26149"
server_key_version: "encryptionkey@0"
I20260812 03:06:20.286484 9460 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20260812 03:06:20.292600 9486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:20.293819 9460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 03:06:20.294184 9460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "7e3ad42912744246bff592afbd751261"
format_stamp: "Formatted at 2026-08-12 03:06:20 on dist-test-slave-zpvx"
server_key: "7d2e88f59738026c765e654874e5586d"
server_key_iv: "beb8b835e7837320c02c293c3ab26149"
server_key_version: "encryptionkey@0"
I20260812 03:06:20.294588 9460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:20.338464 9460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:20.344674 9460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:20.345162 9460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:20.414804 9460 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:39441
I20260812 03:06:20.414867 9586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:39441 every 8 connection(s)
I20260812 03:06:20.417673 9460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:20.423034 9588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:20.425441 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 9460
I20260812 03:06:20.426090 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:20.427182 10713 external_mini_cluster.cc:1428] Setting key 5704a2dfbd1228465c744f625ecf7247
I20260812 03:06:20.447817 9588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261: Bootstrap starting.
Aug 12 03:06:20 dist-test-slave-zpvx krb5kdc[9435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:20.453524 9588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:20.455812 9588 log.cc:824] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:20.462191 9588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261: No bootstrap required, opened a new log
I20260812 03:06:20.464156 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:20.429379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36741 (local address 127.10.118.126:39441)
0812 03:06:20.432018 (+ 2639us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:20.432059 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:20.432138 (+ 79us) server_negotiation.cc:407] Connection header received
0812 03:06:20.436374 (+ 4236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:20.436453 (+ 79us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:20.436952 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:20.437438 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:20.439889 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:20.441707 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:20.443714 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:20.444661 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:20.444997 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:20.451064 (+ 6067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:20.451089 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:20.451121 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:20.451229 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:20.456427 (+ 5198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:20.458401 (+ 1974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:20.458438 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:20.458469 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:20.458723 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:20.459409 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:20.459440 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:20.459464 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:20.460330 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:20.460902 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:20.461790 (+ 888us) server_negotiation.cc:299] Negotiation successful
0812 03:06:20.462223 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2079,"thread_start_us":1670,"threads_started":1}
I20260812 03:06:20.468878 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:20.486019 9588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ad42912744246bff592afbd751261" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39441 } }
I20260812 03:06:20.487052 9588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:20.487295 9588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3ad42912744246bff592afbd751261, State: Initialized, Role: FOLLOWER
I20260812 03:06:20.488104 9588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ad42912744246bff592afbd751261" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39441 } }
I20260812 03:06:20.488857 9588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:20.489223 9588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:20.489622 9588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:20.495702 9588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ad42912744246bff592afbd751261" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39441 } }
I20260812 03:06:20.496642 9588 leader_election.cc:302] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e3ad42912744246bff592afbd751261; no voters:
I20260812 03:06:20.498467 9588 leader_election.cc:288] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:20.498976 9610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:20.502597 9610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [term 1 LEADER]: Becoming Leader. State: Replica: 7e3ad42912744246bff592afbd751261, State: Running, Role: LEADER
I20260812 03:06:20.503491 9588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:20.503469 9610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ad42912744246bff592afbd751261" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39441 } }
I20260812 03:06:20.509353 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:20.481526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40669 (local address 127.10.118.126:39441)
0812 03:06:20.482345 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:20.482372 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:20.483087 (+ 715us) server_negotiation.cc:407] Connection header received
0812 03:06:20.483457 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:20.483477 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:20.483644 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:20.483794 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:20.487104 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:20.488116 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:20.490848 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:20.491430 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:20.491621 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:20.496155 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:20.496206 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:20.496240 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:20.496384 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:20.501733 (+ 5349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:20.503219 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:20.503249 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:20.503279 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:20.503541 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:20.504343 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:20.504392 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:20.504413 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:20.505102 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:20.505456 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:20.506300 (+ 844us) server_negotiation.cc:299] Negotiation successful
0812 03:06:20.506753 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260812 03:06:20.514967 9612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e3ad42912744246bff592afbd751261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ad42912744246bff592afbd751261" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39441 } } }
I20260812 03:06:20.515367 9613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e3ad42912744246bff592afbd751261. Latest consensus state: current_term: 1 leader_uuid: "7e3ad42912744246bff592afbd751261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e3ad42912744246bff592afbd751261" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39441 } } }
I20260812 03:06:20.515889 9612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:20.516314 9613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:20.531349 9620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:20.543056 9620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:20.572461 9620 catalog_manager.cc:1347] Generated new cluster ID: 4c28b88e9d2543f3aa9203e958497f26
I20260812 03:06:20.577831 9620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:20.605003 9620 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:20.607628 9620 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:20.640061 9620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e3ad42912744246bff592afbd751261: Generated new TSK 0
I20260812 03:06:20.641294 9620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:20.661401 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 9460
2026-08-12T03:06:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (3678 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:20 dist-test-slave-zpvx krb5kdc[9644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:20 dist-test-slave-zpvx krb5kdc[9644](info): set up 2 sockets
Aug 12 03:06:20 dist-test-slave-zpvx krb5kdc[9644](info): commencing operation
krb5kdc: starting...
W20260812 03:06:22.061324 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:22 dist-test-slave-zpvx krb5kdc[9644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:22.309350 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:45075
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:39925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:22.620307 9670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:22.620920 9670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:22.621503 9670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:22.652786 9670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:22.653143 9670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:22.653389 9670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:22.653587 9670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:22.653772 9670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:22.694581 9670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:39925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:22.695739 9670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:22.696823 9670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:22.714073 9682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:22.714078 9683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:22.715036 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:23.909713 9686 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:06:23.909890 9670 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:06:23.913486 9670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:23.916704 9670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:23.918149 9670 hybrid_clock.cc:648] HybridClock initialized: now 1786503983918101 us; error 73 us; skew 500 ppm
I20260812 03:06:23.918929 9670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:23 dist-test-slave-zpvx krb5kdc[9644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503983, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:23.929168 9670 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:23.934603 9670 webserver.cc:460] Webserver started at http://127.10.118.126:35875/ using document root <none> and password file <none>
I20260812 03:06:23.935534 9670 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:23.935755 9670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:23.936235 9670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:23.941526 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "6ba96ad35ac84740b185a968733f7328"
format_stamp: "Formatted at 2026-08-12 03:06:23 on dist-test-slave-zpvx"
server_key: "218206e3a87454aa12f1000e6db6fab8"
server_key_iv: "b2d654a036b3f3f5f7426fb0623ffaad"
server_key_version: "encryptionkey@0"
I20260812 03:06:23.942689 9670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "6ba96ad35ac84740b185a968733f7328"
format_stamp: "Formatted at 2026-08-12 03:06:23 on dist-test-slave-zpvx"
server_key: "218206e3a87454aa12f1000e6db6fab8"
server_key_iv: "b2d654a036b3f3f5f7426fb0623ffaad"
server_key_version: "encryptionkey@0"
I20260812 03:06:23.949935 9670 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260812 03:06:23.955350 9703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:23.956439 9670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 03:06:23.956759 9670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "6ba96ad35ac84740b185a968733f7328"
format_stamp: "Formatted at 2026-08-12 03:06:23 on dist-test-slave-zpvx"
server_key: "218206e3a87454aa12f1000e6db6fab8"
server_key_iv: "b2d654a036b3f3f5f7426fb0623ffaad"
server_key_version: "encryptionkey@0"
I20260812 03:06:23.957149 9670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:23.995841 9670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:24.001670 9670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:24.002158 9670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:24.070214 9670 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:45075
I20260812 03:06:24.070324 9797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:45075 every 8 connection(s)
I20260812 03:06:24.072917 9670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:24.077654 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 9670
I20260812 03:06:24.078544 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:24.079022 9799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:24.080205 10713 external_mini_cluster.cc:1428] Setting key 0ba82cc9825e7e8038db2a24479cd092
I20260812 03:06:24.103813 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:24.082992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40781 (local address 127.10.118.126:45075)
0812 03:06:24.085776 (+ 2784us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:24.085802 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:24.085855 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:06:24.090096 (+ 4241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:24.090135 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:24.090512 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:24.090842 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:24.092503 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:24.094132 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:24.097130 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:24.097984 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:24.098218 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:24.100255 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:24.100286 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:24.100312 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:24.101123 (+ 811us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:24.101241 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:24.101316 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:24.101436 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:24.101816 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 03:06:24.102048 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2359,"thread_start_us":1855,"threads_started":1}
I20260812 03:06:24.105790 9799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328: Bootstrap starting.
I20260812 03:06:24.108445 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:24.113124 9799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:24.115497 9799 log.cc:824] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:24.121572 9799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328: No bootstrap required, opened a new log
I20260812 03:06:24.141436 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:24.129420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48213 (local address 127.10.118.126:45075)
0812 03:06:24.129951 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:24.129979 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:24.131015 (+ 1036us) server_negotiation.cc:407] Connection header received
0812 03:06:24.131302 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:24.131329 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:24.131564 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:24.131770 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:24.132770 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:24.134153 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:24.136007 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:24.136950 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:24.137248 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:24.138732 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:24.138763 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:24.138793 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:24.138855 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:24.138965 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:24.139036 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:24.139195 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:24.139440 (+ 245us) server_negotiation.cc:299] Negotiation successful
0812 03:06:24.139644 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 03:06:24.146736 9799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba96ad35ac84740b185a968733f7328" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 45075 } }
I20260812 03:06:24.147387 9799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:24.147591 9799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba96ad35ac84740b185a968733f7328, State: Initialized, Role: FOLLOWER
I20260812 03:06:24.148206 9799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba96ad35ac84740b185a968733f7328" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 45075 } }
I20260812 03:06:24.148778 9799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:24.148998 9799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:24.149286 9799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:24.153198 9799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba96ad35ac84740b185a968733f7328" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 45075 } }
I20260812 03:06:24.153831 9799 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba96ad35ac84740b185a968733f7328; no voters:
I20260812 03:06:24.155591 9799 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:24.156061 9811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:24.158566 9811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [term 1 LEADER]: Becoming Leader. State: Replica: 6ba96ad35ac84740b185a968733f7328, State: Running, Role: LEADER
I20260812 03:06:24.159299 9811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba96ad35ac84740b185a968733f7328" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 45075 } }
I20260812 03:06:24.160259 9799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:24.171101 9814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba96ad35ac84740b185a968733f7328. Latest consensus state: current_term: 1 leader_uuid: "6ba96ad35ac84740b185a968733f7328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba96ad35ac84740b185a968733f7328" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 45075 } } }
I20260812 03:06:24.171844 9814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:24.173450 9812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba96ad35ac84740b185a968733f7328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba96ad35ac84740b185a968733f7328" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 45075 } } }
I20260812 03:06:24.174374 9812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:24.176250 9831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:24.188220 9831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:24.206223 9831 catalog_manager.cc:1347] Generated new cluster ID: c702c154ad6f4fabb7fa022af9da1ca7
I20260812 03:06:24.206519 9831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:24.222585 9831 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:24.224658 9831 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:24.236169 9831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ba96ad35ac84740b185a968733f7328: Generated new TSK 0
I20260812 03:06:24.237155 9831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:24.251130 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 9670
2026-08-12T03:06:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (3587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:24 dist-test-slave-zpvx krb5kdc[9851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:24 dist-test-slave-zpvx krb5kdc[9851](info): set up 2 sockets
Aug 12 03:06:24 dist-test-slave-zpvx krb5kdc[9851](info): commencing operation
krb5kdc: starting...
W20260812 03:06:25.641288 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.328s user 0.007s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:25 dist-test-slave-zpvx krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:25.888509 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44937
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:36145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:44937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:26.201378 9874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:26.201953 9874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:26.202540 9874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:26.233125 9874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:26.233471 9874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:26.233711 9874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:26.233904 9874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:26.234092 9874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:26.270048 9874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:36145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:44937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:26.271175 9874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:26.272181 9874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:26.289084 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:26.289511 9889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:26.291527 9874 server_base.cc:1034] running on GCE node
W20260812 03:06:26.289285 9891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:27.493999 9874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:27.497628 9874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:27.499080 9874 hybrid_clock.cc:648] HybridClock initialized: now 1786503987499053 us; error 46 us; skew 500 ppm
I20260812 03:06:27.500113 9874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:27 dist-test-slave-zpvx krb5kdc[9851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503987, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:27.506354 9874 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:27.511610 9874 webserver.cc:460] Webserver started at http://127.10.118.126:45009/ using document root <none> and password file <none>
I20260812 03:06:27.512631 9874 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:27.512861 9874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:27.513360 9874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:27.518694 9874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "494b6c4b2bee491886890ea7cc5dcff8"
format_stamp: "Formatted at 2026-08-12 03:06:27 on dist-test-slave-zpvx"
server_key: "d8f4874a8ec1d0b7173da0c57b3a05f5"
server_key_iv: "386f1017414328e7771aa902934ce480"
server_key_version: "encryptionkey@0"
I20260812 03:06:27.519899 9874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "494b6c4b2bee491886890ea7cc5dcff8"
format_stamp: "Formatted at 2026-08-12 03:06:27 on dist-test-slave-zpvx"
server_key: "d8f4874a8ec1d0b7173da0c57b3a05f5"
server_key_iv: "386f1017414328e7771aa902934ce480"
server_key_version: "encryptionkey@0"
I20260812 03:06:27.527215 9874 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 03:06:27.532769 9913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:27.533855 9874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 03:06:27.534193 9874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "494b6c4b2bee491886890ea7cc5dcff8"
format_stamp: "Formatted at 2026-08-12 03:06:27 on dist-test-slave-zpvx"
server_key: "d8f4874a8ec1d0b7173da0c57b3a05f5"
server_key_iv: "386f1017414328e7771aa902934ce480"
server_key_version: "encryptionkey@0"
I20260812 03:06:27.534581 9874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:27.560098 9874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:27.565768 9874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:27.566232 9874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:27.636194 9874 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:44937
I20260812 03:06:27.636268 10009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:44937 every 8 connection(s)
I20260812 03:06:27.639029 9874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:27.644310 10012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:27.646180 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 9874
I20260812 03:06:27.646657 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:27.647820 10713 external_mini_cluster.cc:1428] Setting key f2dead60a4ebfa9d3d178aef51102fdf
I20260812 03:06:27.663229 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:27.650000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45929 (local address 127.10.118.126:44937)
0812 03:06:27.652619 (+ 2619us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:27.652659 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:27.652733 (+ 74us) server_negotiation.cc:407] Connection header received
0812 03:06:27.657544 (+ 4811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:27.657603 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:27.658044 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:27.658530 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:27.660026 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:27.660060 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:27.660102 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:27.660160 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:27.660300 (+ 140us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:27.660402 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:27.660688 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:27.660889 (+ 201us) server_negotiation.cc:299] Negotiation successful
0812 03:06:27.661116 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2036,"thread_start_us":1607,"threads_started":1}
I20260812 03:06:27.667649 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:27.671089 10012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8: Bootstrap starting.
I20260812 03:06:27.676826 10012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:27.678575 10012 log.cc:824] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:27.685187 10012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8: No bootstrap required, opened a new log
I20260812 03:06:27.687345 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:27.681099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40471 (local address 127.10.118.126:44937)
0812 03:06:27.681739 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:27.681780 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:27.682967 (+ 1187us) server_negotiation.cc:407] Connection header received
0812 03:06:27.683271 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:27.683298 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:27.683485 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:27.683683 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:27.685093 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:27.685121 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:27.685150 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 03:06:27.685214 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:27.685330 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 03:06:27.685404 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:27.685646 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:27.685795 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 03:06:27.685928 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260812 03:06:27.707052 10012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b6c4b2bee491886890ea7cc5dcff8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44937 } }
I20260812 03:06:27.707763 10012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 03:06:27.707975 10012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494b6c4b2bee491886890ea7cc5dcff8, State: Initialized, Role: FOLLOWER
I20260812 03:06:27.708698 10012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b6c4b2bee491886890ea7cc5dcff8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44937 } }
I20260812 03:06:27.709267 10012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 03:06:27.709503 10012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 03:06:27.709758 10012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:27.714222 10012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b6c4b2bee491886890ea7cc5dcff8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44937 } }
I20260812 03:06:27.714854 10012 leader_election.cc:302] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 494b6c4b2bee491886890ea7cc5dcff8; no voters:
I20260812 03:06:27.716485 10012 leader_election.cc:288] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 03:06:27.716884 10033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:27.720144 10033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [term 1 LEADER]: Becoming Leader. State: Replica: 494b6c4b2bee491886890ea7cc5dcff8, State: Running, Role: LEADER
I20260812 03:06:27.720909 10012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:27.720912 10033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b6c4b2bee491886890ea7cc5dcff8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44937 } }
I20260812 03:06:27.734684 10035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "494b6c4b2bee491886890ea7cc5dcff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b6c4b2bee491886890ea7cc5dcff8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44937 } } }
I20260812 03:06:27.734831 10036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494b6c4b2bee491886890ea7cc5dcff8. Latest consensus state: current_term: 1 leader_uuid: "494b6c4b2bee491886890ea7cc5dcff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "494b6c4b2bee491886890ea7cc5dcff8" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 44937 } } }
I20260812 03:06:27.735607 10035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:27.735625 10036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:27.739853 10049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 03:06:27.751694 10049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:27.767467 10049 catalog_manager.cc:1347] Generated new cluster ID: 6582fe28fa8e475ba0622658cb112cd0
I20260812 03:06:27.767746 10049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:27.788941 10049 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:27.789588 10049 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:27.809484 10049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 494b6c4b2bee491886890ea7cc5dcff8: Generated new TSK 0
I20260812 03:06:27.810437 10049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:27.825559 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 9874
2026-08-12T03:06:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (3575 ms)
[----------] 18 tests from AuthTokenIssuingTest (64560 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:27 dist-test-slave-zpvx krb5kdc[10068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:27 dist-test-slave-zpvx krb5kdc[10068](info): set up 2 sockets
Aug 12 03:06:27 dist-test-slave-zpvx krb5kdc[10068](info): commencing operation
krb5kdc: starting...
W20260812 03:06:29.206826 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.317s user 0.008s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:29 dist-test-slave-zpvx krb5kdc[10068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:29.450192 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:39375
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:39375,127.10.118.125:43283,127.10.118.124:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:29.759431 10095 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 03:06:29.760294 10095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:29.760681 10095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:29.761327 10095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:29.791750 10095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:29.792140 10095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:29.792467 10095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:29.792737 10095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:29.792991 10095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:29.828604 10095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:39375,127.10.118.125:43283,127.10.118.124:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:39375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:29.829869 10095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:29.830938 10095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:29.848196 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:29.848240 10118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:29.848781 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:31.050120 10117 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:06:31.050288 10095 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:06:31.055018 10095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:31.058352 10095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:31.059803 10095 hybrid_clock.cc:648] HybridClock initialized: now 1786503991059756 us; error 67 us; skew 500 ppm
I20260812 03:06:31.060668 10095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:31 dist-test-slave-zpvx krb5kdc[10068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:31.066620 10095 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:31.072474 10095 webserver.cc:460] Webserver started at http://127.10.118.126:42665/ using document root <none> and password file <none>
I20260812 03:06:31.073403 10095 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:31.073642 10095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:31.074120 10095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:31.079490 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "987a845b13194a52b77a487eddffd513"
format_stamp: "Formatted at 2026-08-12 03:06:31 on dist-test-slave-zpvx"
server_key: "e147afd492e64cc5114e65d88a629ccd"
server_key_iv: "8180ec975860583d3dd1b3114902df51"
server_key_version: "encryptionkey@0"
I20260812 03:06:31.080715 10095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "987a845b13194a52b77a487eddffd513"
format_stamp: "Formatted at 2026-08-12 03:06:31 on dist-test-slave-zpvx"
server_key: "e147afd492e64cc5114e65d88a629ccd"
server_key_iv: "8180ec975860583d3dd1b3114902df51"
server_key_version: "encryptionkey@0"
I20260812 03:06:31.088055 10095 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 03:06:31.093623 10129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:31.094655 10095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260812 03:06:31.094996 10095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "987a845b13194a52b77a487eddffd513"
format_stamp: "Formatted at 2026-08-12 03:06:31 on dist-test-slave-zpvx"
server_key: "e147afd492e64cc5114e65d88a629ccd"
server_key_iv: "8180ec975860583d3dd1b3114902df51"
server_key_version: "encryptionkey@0"
I20260812 03:06:31.095393 10095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:31.131794 10095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:31.137423 10095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:31.137905 10095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:31.206537 10095 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:39375
I20260812 03:06:31.206637 10222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:39375 every 8 connection(s)
I20260812 03:06:31.209412 10095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:31.214658 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:31.219267 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 10095
I20260812 03:06:31.219755 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:31.221041 10713 external_mini_cluster.cc:1428] Setting key cb6d85feb8cc66ef3b644ff2a048b6e7
I20260812 03:06:31.226267 10224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.118.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.125@KRBTEST.COM" created.
Aug 12 03:06:31 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:31.267906 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.234302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.10.118.126:39375)
0812 03:06:31.237195 (+ 2893us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:31.237231 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:31.237313 (+ 82us) server_negotiation.cc:407] Connection header received
0812 03:06:31.239800 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:31.239839 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:31.240314 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:31.240640 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:31.242933 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:31.245111 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:31.247844 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:31.248716 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:31.248956 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:31.255128 (+ 6172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:31.255153 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:31.255183 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:31.255292 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:31.259945 (+ 4653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:31.261153 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:31.261175 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:31.261188 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:31.261429 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:31.262106 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:31.262127 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:31.262140 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:31.262875 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:31.263175 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:31.264167 (+ 992us) server_negotiation.cc:299] Negotiation successful
0812 03:06:31.264931 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":261,"server-negotiator.queue_time_us":2300,"thread_start_us":1402,"threads_started":1}
I20260812 03:06:31.267959 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.231314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:39375 (local address 127.0.0.1:42238)
0812 03:06:31.234031 (+ 2717us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.234129 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:31.239171 (+ 5042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:31.240696 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:31.240718 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:31.241842 (+ 1124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:31.242551 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:31.242596 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:31.245355 (+ 2759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:31.245376 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:31.247525 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:31.247583 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:31.249094 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:31.249142 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:31.249669 (+ 527us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:31.250747 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:31.250851 (+ 104us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:31.254742 (+ 3891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:31.260166 (+ 5424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:31.260204 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:31.260224 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:31.260942 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:31.261628 (+ 686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:31.261647 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:31.261661 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:31.261914 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:31.263090 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:31.263145 (+ 55us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:31.263526 (+ 381us) client_negotiation.cc:769] Sending connection context
0812 03:06:31.263892 (+ 366us) client_negotiation.cc:240] Negotiation successful
0812 03:06:31.264660 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2161,"thread_start_us":1730,"threads_started":1}
I20260812 03:06:31.271209 10224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } has no permanent_uuid. Determining permanent_uuid...
I20260812 03:06:31.273518 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.272305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39658)
0812 03:06:31.272795 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.273017 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260812 03:06:31.274044 10137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab.
Entry for principal kudu/127.10.118.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab.
I20260812 03:06:31.278294 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.277024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39664)
0812 03:06:31.277542 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.277742 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260812 03:06:31.278765 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:43283: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
I20260812 03:06:31.279778 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.125:43283
--webserver_interface=127.10.118.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab
--principal=kudu/127.10.118.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:39375,127.10.118.125:43283,127.10.118.124:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 03:06:31.328310 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } attempt: 1
I20260812 03:06:31.330683 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.329683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39668)
0812 03:06:31.330124 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.330286 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 03:06:31.333036 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.331996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39672)
0812 03:06:31.332508 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.332656 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260812 03:06:31.333474 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:43283: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
I20260812 03:06:31.401945 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } attempt: 2
I20260812 03:06:31.404203 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.403170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39680)
0812 03:06:31.403641 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.403803 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 03:06:31.406468 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.405539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39686)
0812 03:06:31.405963 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.406101 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 03:06:31.406868 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:43283: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
I20260812 03:06:31.498363 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } attempt: 3
I20260812 03:06:31.500730 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.499600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39688)
0812 03:06:31.500089 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.500250 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260812 03:06:31.503440 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.502295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39698)
0812 03:06:31.502849 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.503052 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260812 03:06:31.503960 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:43283: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
W20260812 03:06:31.589334 10237 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 03:06:31.590195 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:31.590549 10237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:31.591172 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:31.621718 10237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:31.622087 10237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:31.622397 10237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:31.622654 10237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:31.622895 10237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:31.647346 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } attempt: 4
I20260812 03:06:31.649607 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.648547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39710)
0812 03:06:31.649015 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.649208 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 03:06:31.651814 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.650865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39712)
0812 03:06:31.651303 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.651443 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260812 03:06:31.652213 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:43283: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
I20260812 03:06:31.659371 10237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:39375,127.10.118.125:43283,127.10.118.124:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab
--principal=kudu/127.10.118.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.125:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.118.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:31.660622 10237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:31.661693 10237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:31.680357 10251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:31.951704 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } attempt: 5
I20260812 03:06:31.955639 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.953588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39722)
0812 03:06:31.954407 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.954738 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260812 03:06:31.959551 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:31.957676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39728)
0812 03:06:31.958391 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:31.958701 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260812 03:06:31.960189 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:43283: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
I20260812 03:06:32.507781 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } attempt: 6
I20260812 03:06:32.515705 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:32.509598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39734)
0812 03:06:32.513916 (+ 4318us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:32.514387 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3636,"thread_start_us":3166,"threads_started":1}
I20260812 03:06:32.524696 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:32.522700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39740)
0812 03:06:32.523473 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:32.523800 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260812 03:06:32.525312 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:43283: Network error: Client connection negotiation failed: client connection to 127.10.118.125:43283: connect: Connection refused (error 111)
W20260812 03:06:31.680591 10255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:31.680395 10252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:32.918581 10254 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1237 milliseconds
I20260812 03:06:32.918722 10237 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:06:32.920073 10237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:32.922705 10237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:32.924048 10237 hybrid_clock.cc:648] HybridClock initialized: now 1786503992924016 us; error 58 us; skew 500 ppm
I20260812 03:06:32.924909 10237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:32 dist-test-slave-zpvx krb5kdc[10068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503992, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:32.930727 10237 init.cc:377] Logged in from keytab as kudu/127.10.118.125@KRBTEST.COM (short username kudu)
I20260812 03:06:32.935943 10237 webserver.cc:460] Webserver started at http://127.10.118.125:42181/ using document root <none> and password file <none>
I20260812 03:06:32.936887 10237 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:32.937067 10237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:32.937463 10237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:32.942586 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data/instance:
uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a"
format_stamp: "Formatted at 2026-08-12 03:06:32 on dist-test-slave-zpvx"
server_key: "b44f800f2bc93b864a9e6afbd9dfa44e"
server_key_iv: "429485b72b838b57d789e75723af7b24"
server_key_version: "encryptionkey@0"
I20260812 03:06:32.943701 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/wal/instance:
uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a"
format_stamp: "Formatted at 2026-08-12 03:06:32 on dist-test-slave-zpvx"
server_key: "b44f800f2bc93b864a9e6afbd9dfa44e"
server_key_iv: "429485b72b838b57d789e75723af7b24"
server_key_version: "encryptionkey@0"
I20260812 03:06:32.951025 10237 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 03:06:32.956552 10273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:32.957578 10237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260812 03:06:32.957862 10237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/wal
uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a"
format_stamp: "Formatted at 2026-08-12 03:06:32 on dist-test-slave-zpvx"
server_key: "b44f800f2bc93b864a9e6afbd9dfa44e"
server_key_iv: "429485b72b838b57d789e75723af7b24"
server_key_version: "encryptionkey@0"
I20260812 03:06:32.958184 10237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:32.992516 10237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:32.998214 10237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:32.998634 10237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:33.071003 10237 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.125:43283
I20260812 03:06:33.071106 10367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.125:43283 every 8 connection(s)
I20260812 03:06:33.073673 10237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/data/info.pb
I20260812 03:06:33.078830 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 10237
I20260812 03:06:33.079169 10372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:33.079331 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-1/wal/instance
I20260812 03:06:33.080715 10713 external_mini_cluster.cc:1428] Setting key 9e65aa2501e311ac60b440d1f3f58e64
I20260812 03:06:33.091804 10372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.118.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.124@KRBTEST.COM" created.
Aug 12 03:06:33 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503992, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:33.131969 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.097422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.10.118.126:39375)
0812 03:06:33.099578 (+ 2156us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:33.099601 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:33.100593 (+ 992us) server_negotiation.cc:407] Connection header received
0812 03:06:33.107696 (+ 7103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:33.107730 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:33.107932 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:33.108079 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:33.110836 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.112311 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.114439 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.115567 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.115910 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:33.121462 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:33.121484 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:33.121503 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:33.121603 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:33.125282 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:33.126865 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:33.126885 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:33.126899 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:33.127092 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:33.127961 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:33.127983 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:33.127996 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:33.128567 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:33.128815 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:33.130243 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0812 03:06:33.130569 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1608,"thread_start_us":1272,"threads_started":1}
I20260812 03:06:33.134205 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.097505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:39375 (local address 127.0.0.1:42246)
0812 03:06:33.100284 (+ 2779us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.100413 (+ 129us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:33.107274 (+ 6861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:33.108319 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:33.108342 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:33.109324 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:33.110503 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:33.110546 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.112619 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.112641 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:33.114190 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:33.114232 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.115902 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.115923 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:33.116439 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:33.117418 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:33.117484 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:33.121071 (+ 3587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:33.125507 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:33.125546 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:33.125592 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:33.126650 (+ 1058us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:33.127301 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:33.127337 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:33.127368 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:33.127761 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:33.128789 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:33.128840 (+ 51us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:33.129555 (+ 715us) client_negotiation.cc:769] Sending connection context
0812 03:06:33.130065 (+ 510us) client_negotiation.cc:240] Negotiation successful
0812 03:06:33.130891 (+ 826us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2126,"thread_start_us":1726,"threads_started":1}
I20260812 03:06:33.136561 10372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab.
Entry for principal kudu/127.10.118.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab.
I20260812 03:06:33.145949 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.124:46727
--webserver_interface=127.10.118.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab
--principal=kudu/127.10.118.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:39375,127.10.118.125:43283,127.10.118.124:46727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 03:06:33 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503992, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
I20260812 03:06:33.165313 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.138261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.10.118.125:43283)
0812 03:06:33.140685 (+ 2424us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:33.140707 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:33.140752 (+ 45us) server_negotiation.cc:407] Connection header received
0812 03:06:33.140937 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:33.140966 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:33.141199 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:33.141455 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:33.143098 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.145664 (+ 2566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.147331 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.148752 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.149204 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:33.153479 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:33.153523 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:33.153570 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:33.153731 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:33.158225 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:33.159650 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:33.159676 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:33.159689 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:33.159888 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:33.160694 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:33.160714 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:33.160727 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:33.161290 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:33.161498 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:33.162531 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0812 03:06:33.163178 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":379,"server-negotiator.queue_time_us":2012,"spinlock_wait_cycles":24448,"thread_start_us":1301,"threads_started":1}
I20260812 03:06:33.165977 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.137827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39750)
0812 03:06:33.138337 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.138371 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:33.138572 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:33.141499 (+ 2927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:33.141525 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:33.142202 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:33.142715 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:33.142773 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.145892 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.145913 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:33.147075 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:33.147115 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.148999 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.149022 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:33.149554 (+ 532us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:33.150314 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:33.150375 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:33.153112 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:33.158470 (+ 5358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:33.158506 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:33.158537 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:33.159422 (+ 885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:33.160111 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:33.160150 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:33.160182 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:33.160503 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:33.161519 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:33.161557 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:33.161955 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 03:06:33.162336 (+ 381us) client_negotiation.cc:240] Negotiation successful
0812 03:06:33.162949 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":164}
I20260812 03:06:33.169011 10372 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
I20260812 03:06:33.171217 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.170272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53692)
0812 03:06:33.170664 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.170845 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260812 03:06:33.171722 10282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.175981 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.175114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53706)
0812 03:06:33.175483 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.175622 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 03:06:33.176354 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.225869 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 1
I20260812 03:06:33.228045 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.227072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53718)
0812 03:06:33.227501 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.227657 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 03:06:33.230216 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.229355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53726)
0812 03:06:33.229710 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.229853 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 03:06:33.230602 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.299046 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 2
I20260812 03:06:33.301312 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.300236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53730)
0812 03:06:33.300742 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.300905 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260812 03:06:33.303493 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.302556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53744)
0812 03:06:33.302976 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.303115 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 03:06:33.303886 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.395411 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 3
I20260812 03:06:33.397568 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.396521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53756)
0812 03:06:33.397000 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.397159 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 03:06:33.399756 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.398814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53770)
0812 03:06:33.399239 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.399387 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260812 03:06:33.400177 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
W20260812 03:06:33.467373 10383 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 03:06:33.468143 10383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:33.468515 10383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:33.469110 10383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:33.499997 10383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:33.500314 10383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:33.500628 10383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:33.500852 10383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:33.501111 10383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:33.538563 10383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:39375,127.10.118.125:43283,127.10.118.124:46727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab
--principal=kudu/127.10.118.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.124:46727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.118.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:33.539734 10383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:33.540752 10383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 03:06:33.543658 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 4
I20260812 03:06:33.545938 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.544875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53786)
0812 03:06:33.545387 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.545544 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 03:06:33.548920 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.547424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53788)
0812 03:06:33.548035 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.548260 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260812 03:06:33.549463 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
W20260812 03:06:33.558982 10396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:33.585843 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } attempt: 7
Aug 12 03:06:33 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
I20260812 03:06:33.613988 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.587233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39754 (local address 127.10.118.125:43283)
0812 03:06:33.587650 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:33.587668 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:33.589609 (+ 1941us) server_negotiation.cc:407] Connection header received
0812 03:06:33.589949 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:33.589989 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:33.590351 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:33.590621 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:33.592034 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.594058 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.595908 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.597307 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.597722 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:33.603281 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:33.603305 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:33.603322 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:33.603418 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:33.607023 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:33.608941 (+ 1918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:33.608962 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:33.608977 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:33.609186 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:33.610164 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:33.610183 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:33.610197 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:33.610701 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:33.610869 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:33.612202 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0812 03:06:33.612593 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 03:06:33.616773 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.587301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39754)
0812 03:06:33.589418 (+ 2117us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.589455 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:33.589670 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:33.590668 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:33.590687 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:33.591370 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:33.591738 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:33.591773 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.594338 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.594359 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:33.595623 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:33.595657 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:33.597654 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:33.597675 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:33.597937 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:33.598849 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:33.598910 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:33.602905 (+ 3995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:33.607306 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:33.607370 (+ 64us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:33.607408 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:33.608697 (+ 1289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:33.609432 (+ 735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:33.609490 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:33.609527 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:33.609932 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:33.610946 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:33.610997 (+ 51us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:33.611592 (+ 595us) client_negotiation.cc:769] Sending connection context
0812 03:06:33.612017 (+ 425us) client_negotiation.cc:240] Negotiation successful
0812 03:06:33.612908 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1771,"thread_start_us":1459,"threads_started":1}
I20260812 03:06:33.618896 10224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
I20260812 03:06:33.621923 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.620080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53802)
0812 03:06:33.620809 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.621112 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 03:06:33.626737 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.624988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53806)
0812 03:06:33.625655 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.625939 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260812 03:06:33.627260 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.685672 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 1
I20260812 03:06:33.688710 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.686734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53810)
0812 03:06:33.687505 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.687821 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260812 03:06:33.691932 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.690186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53826)
0812 03:06:33.690846 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.691145 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260812 03:06:33.692492 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.773926 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 2
I20260812 03:06:33.776675 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.775455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53828)
0812 03:06:33.776015 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.776181 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260812 03:06:33.778971 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.777949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53834)
0812 03:06:33.778447 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.778591 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260812 03:06:33.779507 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.848894 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 5
I20260812 03:06:33.851050 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.850005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53846)
0812 03:06:33.850489 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.850649 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 03:06:33.853716 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.852785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53854)
0812 03:06:33.853196 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.853347 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260812 03:06:33.854130 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:33.864890 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 3
I20260812 03:06:33.867456 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.866409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53862)
0812 03:06:33.866863 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.867024 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 03:06:33.869817 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:33.868839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53872)
0812 03:06:33.869301 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:33.869446 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260812 03:06:33.870224 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:34.010839 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 4
I20260812 03:06:34.013559 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.012494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53888)
0812 03:06:34.012969 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.013129 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 03:06:34.015892 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.014867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53898)
0812 03:06:34.015305 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.015454 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260812 03:06:34.016327 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:34.299962 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 5
I20260812 03:06:34.302637 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.301570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53904)
0812 03:06:34.302048 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.302209 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 03:06:34.304898 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.303875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53910)
0812 03:06:34.304319 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.304512 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260812 03:06:34.305330 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:34.401747 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 6
I20260812 03:06:34.406059 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.403303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53926)
0812 03:06:34.405391 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.405558 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1733,"thread_start_us":1465,"threads_started":1}
I20260812 03:06:34.409883 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.408900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53928)
0812 03:06:34.409353 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.409494 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260812 03:06:34.410302 10372 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
W20260812 03:06:33.558990 10399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:33.559089 10395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:33.560222 10383 server_base.cc:1034] running on GCE node
I20260812 03:06:34.769775 10383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:34.772588 10383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:34.773967 10383 hybrid_clock.cc:648] HybridClock initialized: now 1786503994773927 us; error 60 us; skew 500 ppm
I20260812 03:06:34.774842 10383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:34 dist-test-slave-zpvx krb5kdc[10068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503994, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:34.781261 10383 init.cc:377] Logged in from keytab as kudu/127.10.118.124@KRBTEST.COM (short username kudu)
I20260812 03:06:34.786099 10383 webserver.cc:460] Webserver started at http://127.10.118.124:44443/ using document root <none> and password file <none>
I20260812 03:06:34.787343 10383 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:34.787585 10383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:34.788050 10383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:34.793362 10383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data/instance:
uuid: "43f9dd776aef472ea9cd7c3ef56c9215"
format_stamp: "Formatted at 2026-08-12 03:06:34 on dist-test-slave-zpvx"
server_key: "b923b26d39800074908ee1df2bbe1055"
server_key_iv: "96114424d8bc0588f9c770d1a7d7a649"
server_key_version: "encryptionkey@0"
I20260812 03:06:34.794538 10383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/wal/instance:
uuid: "43f9dd776aef472ea9cd7c3ef56c9215"
format_stamp: "Formatted at 2026-08-12 03:06:34 on dist-test-slave-zpvx"
server_key: "b923b26d39800074908ee1df2bbe1055"
server_key_iv: "96114424d8bc0588f9c770d1a7d7a649"
server_key_version: "encryptionkey@0"
I20260812 03:06:34.801832 10383 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 03:06:34.807446 10430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:34.808478 10383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:06:34.808810 10383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/wal
uuid: "43f9dd776aef472ea9cd7c3ef56c9215"
format_stamp: "Formatted at 2026-08-12 03:06:34 on dist-test-slave-zpvx"
server_key: "b923b26d39800074908ee1df2bbe1055"
server_key_iv: "96114424d8bc0588f9c770d1a7d7a649"
server_key_version: "encryptionkey@0"
I20260812 03:06:34.809188 10383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:34.846021 10383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:34.851847 10383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:34.852344 10383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:34.857918 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 6
I20260812 03:06:34.861937 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.859125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53930)
0812 03:06:34.860896 (+ 1771us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.861142 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1300,"thread_start_us":983,"threads_started":1}
I20260812 03:06:34.865971 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.864985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53946)
0812 03:06:34.865436 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.865582 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260812 03:06:34.866380 10224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:46727: Network error: Client connection negotiation failed: client connection to 127.10.118.124:46727: connect: Connection refused (error 111)
I20260812 03:06:34.925796 10383 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.124:46727
I20260812 03:06:34.925909 10548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.124:46727 every 8 connection(s)
I20260812 03:06:34.928774 10383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/data/info.pb
I20260812 03:06:34.934048 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:34.937312 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 10383
I20260812 03:06:34.938033 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786503542438041-10713-0/minicluster-data/master-2/wal/instance
I20260812 03:06:34.939297 10713 external_mini_cluster.cc:1428] Setting key 9309984713aa2a5ebaa4cbf501943a7f
I20260812 03:06:34.947171 10549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 03:06:34 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
Aug 12 03:06:34 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503994, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:34.979393 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.941969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53956 (local address 127.10.118.124:46727)
0812 03:06:34.944840 (+ 2871us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:34.944866 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:34.944919 (+ 53us) server_negotiation.cc:407] Connection header received
0812 03:06:34.948786 (+ 3867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:34.948828 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:34.949186 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:34.949516 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:34.953262 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.956126 (+ 2864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.959395 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.960269 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.960640 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:34.967678 (+ 7038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:34.967715 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:34.967757 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:34.967902 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:34.972316 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:34.973435 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:34.973465 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:34.973483 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:34.973700 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:34.974357 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:34.974383 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:34.974397 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:34.975052 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:34.975559 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:34.976264 (+ 705us) server_negotiation.cc:299] Negotiation successful
0812 03:06:34.977228 (+ 964us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2456,"thread_start_us":1896,"threads_started":1}
I20260812 03:06:34.981496 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.953558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.10.118.126:39375)
0812 03:06:34.955824 (+ 2266us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:34.955858 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:34.957096 (+ 1238us) server_negotiation.cc:407] Connection header received
0812 03:06:34.957474 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:34.957493 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:34.957668 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:34.957817 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:34.960507 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.961470 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.964329 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.965135 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.965339 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:34.970939 (+ 5600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:34.970961 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:34.970980 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:34.971081 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:34.974750 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:34.975946 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:34.975975 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:34.975988 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:34.976181 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:34.976896 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:34.976916 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:34.976930 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:34.977399 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:34.977643 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:34.978616 (+ 973us) server_negotiation.cc:299] Negotiation successful
0812 03:06:34.979092 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1716,"thread_start_us":1451,"threads_started":1}
I20260812 03:06:34.981874 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.953542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:39375 (local address 127.0.0.1:42252)
0812 03:06:34.956847 (+ 3305us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.956937 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:34.957251 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:34.958009 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:34.958040 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:34.959433 (+ 1393us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:34.960176 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:34.960225 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.961721 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.961742 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:34.964026 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:34.964081 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.965476 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.965520 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:34.965908 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:34.966675 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:34.966738 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:34.970604 (+ 3866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:34.974961 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:34.974994 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:34.975022 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:34.975742 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:34.976398 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:34.976429 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:34.976454 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:34.976681 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:34.977633 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:34.977693 (+ 60us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:34.978070 (+ 377us) client_negotiation.cc:769] Sending connection context
0812 03:06:34.978372 (+ 302us) client_negotiation.cc:240] Negotiation successful
0812 03:06:34.978976 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2746,"mutex_wait_us":44,"thread_start_us":1897,"threads_started":1}
I20260812 03:06:34.985008 10549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 03:06:34 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
Aug 12 03:06:34 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503994, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
I20260812 03:06:35.009723 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.982905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.10.118.125:43283)
0812 03:06:34.984708 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:34.984729 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:34.984796 (+ 67us) server_negotiation.cc:407] Connection header received
0812 03:06:34.984932 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:34.984956 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:34.985187 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:34.985388 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:34.988057 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.989132 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.991005 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.991958 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.992244 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:34.998206 (+ 5962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:34.998237 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:34.998277 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:34.998431 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:35.003082 (+ 4651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.004297 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.004331 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.004353 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.004630 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.005376 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.005409 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.005432 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.005979 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:35.006468 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:35.007145 (+ 677us) server_negotiation.cc:299] Negotiation successful
0812 03:06:35.007644 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":1337,"spinlock_wait_cycles":56064,"thread_start_us":1316,"threads_started":1}
I20260812 03:06:35.009799 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.986463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:43283 (local address 127.0.0.1:39782)
0812 03:06:34.986996 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:34.987045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:34.987288 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:34.989013 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:34.989040 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:34.989645 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:34.989928 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:34.989959 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.991700 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.991729 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:34.993105 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:34.993140 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.994620 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.994640 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:34.994870 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:34.995523 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:34.995581 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:34.998225 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:35.003688 (+ 5463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.003708 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.003722 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.004402 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.005086 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.005105 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.005119 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.005365 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.006361 (+ 996us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:35.006389 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:35.006684 (+ 295us) client_negotiation.cc:769] Sending connection context
0812 03:06:35.006946 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 03:06:35.007475 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260812 03:06:35.010644 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:34.986596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.10.118.125:43283)
0812 03:06:34.988443 (+ 1847us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:34.988459 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:34.988493 (+ 34us) server_negotiation.cc:407] Connection header received
0812 03:06:34.988591 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:34.988608 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:34.988769 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:34.988904 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:34.990196 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.991481 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.993342 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:34.994371 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:34.994673 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:34.998511 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:34.998532 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:34.998548 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:34.998648 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:35.003485 (+ 4837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.004589 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.004614 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.004628 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.004896 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.005598 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.005619 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.005633 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.006128 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:35.006406 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:35.007220 (+ 814us) server_negotiation.cc:299] Negotiation successful
0812 03:06:35.007981 (+ 761us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":301,"server-negotiator.queue_time_us":1463,"thread_start_us":1157,"threads_started":1}
I20260812 03:06:35.012444 10549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 03:06:35 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503994, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
Aug 12 03:06:35 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:35.035614 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:35.013742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:53970)
0812 03:06:35.014200 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:35.014230 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:35.014423 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:35.015250 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:35.015270 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:35.015929 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:35.016318 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:35.016350 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.017678 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.017705 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:35.018729 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:35.018763 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.019941 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.019974 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:35.020435 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:35.021186 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:35.021254 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:35.023898 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:35.029438 (+ 5540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.029459 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.029473 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.030043 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.030787 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.030819 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.030842 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.031117 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.032117 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:35.032141 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:35.032530 (+ 389us) client_negotiation.cc:769] Sending connection context
0812 03:06:35.032851 (+ 321us) client_negotiation.cc:240] Negotiation successful
0812 03:06:35.033300 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":5}
I20260812 03:06:35.036100 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:35.013884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53970 (local address 127.10.118.124:46727)
0812 03:06:35.014688 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:35.014706 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:35.014737 (+ 31us) server_negotiation.cc:407] Connection header received
0812 03:06:35.014835 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:35.014851 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:35.015023 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:35.015147 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:35.016622 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.017476 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.018968 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.019663 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.019950 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:35.024213 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:35.024234 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:35.024254 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:35.024349 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:35.029164 (+ 4815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.030249 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.030280 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.030303 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.030572 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.031307 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.031326 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.031339 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.031911 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:35.032165 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:35.033082 (+ 917us) server_negotiation.cc:299] Negotiation successful
0812 03:06:35.033595 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":477}
I20260812 03:06:35.037285 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:35.012776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42256 (local address 127.10.118.126:39375)
0812 03:06:35.013403 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:35.013433 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:35.013484 (+ 51us) server_negotiation.cc:407] Connection header received
0812 03:06:35.013639 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:35.013667 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:35.013899 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:35.014075 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:35.017264 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.018104 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.019717 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.020301 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.020636 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:35.025911 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:35.025942 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:35.025967 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:35.026106 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:35.030571 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.031811 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.031841 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.031874 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.032124 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.032900 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.032929 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.032951 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.033626 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:35.034041 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:35.034778 (+ 737us) server_negotiation.cc:299] Negotiation successful
0812 03:06:35.035221 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260812 03:06:35.043965 10549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: Bootstrap starting.
I20260812 03:06:35.044516 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:35.049567 10549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:35.051457 10549 log.cc:824] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:35.056545 10549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: No bootstrap required, opened a new log
I20260812 03:06:35.057546 10181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "987a845b13194a52b77a487eddffd513"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42256
I20260812 03:06:35.086701 10549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:35.087602 10549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f9dd776aef472ea9cd7c3ef56c9215, State: Initialized, Role: FOLLOWER
I20260812 03:06:35.088383 10549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:35.091344 10564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:35.092164 10564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:35.093214 10549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 03:06:35.108098 10446 tablet.cc:2369] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 03:06:35.121460 10589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 03:06:35.121788 10589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 03:06:35.116945 10383 master_runner.cc:186] Error getting master registration for 127.10.118.126:39375: OK, instance_id { permanent_uuid: "987a845b13194a52b77a487eddffd513" instance_seqno: 1786503991187659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 03:06:35.141423 10383 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 03:06:35.470896 10372 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 7
Aug 12 03:06:35 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503992, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
I20260812 03:06:35.499508 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:35.473669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.10.118.124:46727)
0812 03:06:35.474233 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:35.474262 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:35.474420 (+ 158us) server_negotiation.cc:407] Connection header received
0812 03:06:35.474827 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:35.474865 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:35.475037 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:35.475222 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:35.477460 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.478500 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.481343 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.481931 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.482226 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:35.487155 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:35.487187 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:35.487213 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:35.487351 (+ 138us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:35.492459 (+ 5108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.493928 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.493973 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.493998 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.494207 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.495092 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.495126 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.495142 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.495652 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:35.495920 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:35.496854 (+ 934us) server_negotiation.cc:299] Negotiation successful
0812 03:06:35.497312 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260812 03:06:35.499818 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:35.472281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:54188)
0812 03:06:35.474235 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:35.474290 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:35.474586 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:35.475303 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:35.475329 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:35.476499 (+ 1170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:35.477165 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:35.477217 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.478904 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.478935 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:35.480996 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:35.481054 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.482251 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.482307 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:35.482678 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:35.483495 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:35.483561 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:35.486764 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:35.492735 (+ 5971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.492765 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.492791 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.493697 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.494526 (+ 829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.494556 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.494577 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.494866 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.495866 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:35.495899 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:35.496260 (+ 361us) client_negotiation.cc:769] Sending connection context
0812 03:06:35.496610 (+ 350us) client_negotiation.cc:240] Negotiation successful
0812 03:06:35.497147 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1503,"thread_start_us":1892,"threads_started":1}
I20260812 03:06:35.510185 10372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: Bootstrap starting.
I20260812 03:06:35.517270 10372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:35.519559 10372 log.cc:824] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:35.524662 10372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: No bootstrap required, opened a new log
I20260812 03:06:35.548578 10372 raft_consensus.cc:348] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:35.549260 10372 raft_consensus.cc:729] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c45486c3ec1349d5a5e7d62b6bb3be1a, State: Initialized, Role: FOLLOWER
I20260812 03:06:35.549989 10372 consensus_queue.cc:260] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:35.552549 10599 sys_catalog.cc:455] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:35.553333 10599 sys_catalog.cc:458] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:35.554319 10372 sys_catalog.cc:564] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [sys.catalog]: configured and running, proceeding with master startup.
W20260812 03:06:35.583163 10622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 03:06:35.583604 10622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 03:06:35.579560 10237 master_runner.cc:186] Error getting master registration for 127.10.118.126:39375: OK, instance_id { permanent_uuid: "987a845b13194a52b77a487eddffd513" instance_seqno: 1786503991187659 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 03:06:35.613128 10237 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 03:06:35.756528 10288 tablet.cc:2369] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:06:35.899820 10224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } attempt: 7
Aug 12 03:06:35 dist-test-slave-zpvx krb5kdc[10068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503991, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
I20260812 03:06:35.958961 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:35.901362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.10.118.124:46727)
0812 03:06:35.901905 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:35.901935 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:35.916667 (+ 14732us) server_negotiation.cc:407] Connection header received
0812 03:06:35.917088 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:35.917117 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:35.917365 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:35.917556 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:35.919382 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.920808 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.927717 (+ 6909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.928800 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.929165 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:35.944429 (+ 15264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:35.944461 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:35.944488 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:35.944635 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:35.949761 (+ 5126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.951804 (+ 2043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.951834 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.951856 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.952110 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:35.953411 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:35.953443 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:35.953466 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:35.954166 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:35.954387 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:35.956302 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0812 03:06:35.956796 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260812 03:06:35.964747 10625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:35.901221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:46727 (local address 127.0.0.1:54202)
0812 03:06:35.916437 (+ 15216us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:35.916492 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:35.916837 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:35.917675 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:35.917703 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:35.918649 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:35.919093 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:35.919144 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.921042 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.921077 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:35.922922 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:35.922976 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:35.932604 (+ 9628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:35.932638 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:35.933026 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:35.939151 (+ 6125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:35.939249 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:35.943947 (+ 4698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:35.950056 (+ 6109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.950121 (+ 65us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.950144 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.951516 (+ 1372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.952533 (+ 1017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:35.952572 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:35.952626 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:35.953074 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:35.954694 (+ 1620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:35.954787 (+ 93us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:35.955421 (+ 634us) client_negotiation.cc:769] Sending connection context
0812 03:06:35.959718 (+ 4297us) client_negotiation.cc:240] Negotiation successful
0812 03:06:35.960748 (+ 1030us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14721,"thread_start_us":15709,"threads_started":1}
I20260812 03:06:35.975425 10224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513: Bootstrap starting.
I20260812 03:06:35.986709 10224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:35.989530 10224 log.cc:824] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:36.000350 10224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513: No bootstrap required, opened a new log
I20260812 03:06:36.019766 10224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:36.020637 10224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987a845b13194a52b77a487eddffd513, State: Initialized, Role: FOLLOWER
I20260812 03:06:36.021373 10224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:36.024497 10628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.026139 10224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:36.026244 10181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 03:06:36.026641 10181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:36.033694 10181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:36.037181 10326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "987a845b13194a52b77a487eddffd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a"
I20260812 03:06:36.037878 10326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:36.044696 10326 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987a845b13194a52b77a487eddffd513 in term 1.
I20260812 03:06:36.046123 10137 leader_election.cc:302] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 987a845b13194a52b77a487eddffd513, c45486c3ec1349d5a5e7d62b6bb3be1a; no voters:
I20260812 03:06:36.025815 10628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:36.051947 10628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:36.053624 10628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [term 1 LEADER]: Becoming Leader. State: Replica: 987a845b13194a52b77a487eddffd513, State: Running, Role: LEADER
I20260812 03:06:36.052999 10181 leader_election.cc:288] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [CANDIDATE]: Term 1 election: Requested vote from peers c45486c3ec1349d5a5e7d62b6bb3be1a (127.10.118.125:43283), 43f9dd776aef472ea9cd7c3ef56c9215 (127.10.118.124:46727)
I20260812 03:06:36.055500 10628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } }
I20260812 03:06:36.075767 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "987a845b13194a52b77a487eddffd513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "43f9dd776aef472ea9cd7c3ef56c9215"
I20260812 03:06:36.076603 10501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:36.083928 10501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 987a845b13194a52b77a487eddffd513 in term 1.
I20260812 03:06:36.105823 10634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987a845b13194a52b77a487eddffd513. Latest consensus state: current_term: 1 leader_uuid: "987a845b13194a52b77a487eddffd513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.114135 10634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: This master's current role is: LEADER
W20260812 03:06:36.126010 10589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 03:06:36.126420 10589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 03:06:36.147449 10651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260812 03:06:36.148490 10141 tablet.cc:2369] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:06:36.211905 10651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:36.215672 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:36.107465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.10.118.126:39375)
0812 03:06:36.117150 (+ 9685us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.117179 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.132404 (+ 15225us) server_negotiation.cc:407] Connection header received
0812 03:06:36.132581 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.132607 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.132842 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.133041 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.138750 (+ 5709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:36.147829 (+ 9079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:36.152500 (+ 4671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:36.153530 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:36.153805 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:36.192812 (+ 39007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:36.192855 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:36.192884 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:36.193708 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:36.199777 (+ 6069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:36.205195 (+ 5418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:36.205232 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:36.205255 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:36.205522 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:36.206397 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:36.206427 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:36.206451 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:36.207325 (+ 874us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:36.207900 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:36.212652 (+ 4752us) server_negotiation.cc:299] Negotiation successful
0812 03:06:36.213182 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9223,"thread_start_us":18050,"threads_started":1}
I20260812 03:06:36.254348 10501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [term 1 FOLLOWER]: Refusing update from remote peer 987a845b13194a52b77a487eddffd513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:06:36.256322 10326 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [term 1 FOLLOWER]: Refusing update from remote peer 987a845b13194a52b77a487eddffd513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:06:36.259603 10628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:06:36.261106 10628 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:06:36.272470 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:36.119739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.10.118.125:43283)
0812 03:06:36.132279 (+ 12540us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.132307 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.144667 (+ 12360us) server_negotiation.cc:407] Connection header received
0812 03:06:36.154690 (+ 10023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.154721 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.155004 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.155210 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.177871 (+ 22661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:36.179590 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:36.188813 (+ 9223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:36.189835 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:36.190087 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:36.212730 (+ 22643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:36.212766 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:36.212793 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:36.212957 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:36.218919 (+ 5962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:36.240599 (+ 21680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:36.240631 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:36.240654 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:36.241022 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:36.243239 (+ 2217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:36.243270 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:36.243294 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:36.244017 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:36.244452 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:36.247761 (+ 3309us) server_negotiation.cc:299] Negotiation successful
0812 03:06:36.248283 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9914,"thread_start_us":10085,"threads_started":1}
I20260812 03:06:36.285842 10552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:36.145660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.10.118.124:46727)
0812 03:06:36.146171 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.146203 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.167311 (+ 21108us) server_negotiation.cc:407] Connection header received
0812 03:06:36.168133 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.168160 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.168432 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.168579 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.185664 (+ 17085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:36.187315 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:36.198775 (+ 11460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:36.199742 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:36.199992 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:36.225853 (+ 25861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:36.225886 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:36.225912 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:36.226058 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:36.231552 (+ 5494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:36.242025 (+ 10473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:36.242061 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:36.242088 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:36.242360 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:36.245509 (+ 3149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:36.245546 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:36.245571 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:36.246282 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:36.246627 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:36.283126 (+ 36499us) server_negotiation.cc:299] Negotiation successful
0812 03:06:36.283682 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260812 03:06:36.333319 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987a845b13194a52b77a487eddffd513. Latest consensus state: current_term: 1 leader_uuid: "987a845b13194a52b77a487eddffd513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.334414 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:36.352182 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987a845b13194a52b77a487eddffd513. Latest consensus state: current_term: 1 leader_uuid: "987a845b13194a52b77a487eddffd513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.361259 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:36.370819 10670 mvcc.cc:205] Tried to move back new op lower bound from 7317520368631758848 to 7317520367981527040. Current Snapshot: MvccSnapshot[applied={T|T < 7317520368631758848}]
I20260812 03:06:36.384542 10678 mvcc.cc:205] Tried to move back new op lower bound from 7317520368631758848 to 7317520367981527040. Current Snapshot: MvccSnapshot[applied={T|T < 7317520368631758848}]
I20260812 03:06:36.415890 10628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "987a845b13194a52b77a487eddffd513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.416832 10628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:36.418627 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "987a845b13194a52b77a487eddffd513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.419377 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:36.481456 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "987a845b13194a52b77a487eddffd513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.483023 10651 catalog_manager.cc:1347] Generated new cluster ID: 29da43d128b04c4a857894a8fa7d0b82
I20260812 03:06:36.482786 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "987a845b13194a52b77a487eddffd513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "987a845b13194a52b77a487eddffd513" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 39375 } } peers { permanent_uuid: "c45486c3ec1349d5a5e7d62b6bb3be1a" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 43283 } } peers { permanent_uuid: "43f9dd776aef472ea9cd7c3ef56c9215" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 46727 } } }
I20260812 03:06:36.483503 10651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:36.483523 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:36.495136 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:36.564761 10651 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:36.568691 10651 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:36.589246 10622 catalog_manager.cc:1259] Loaded cluster ID: 29da43d128b04c4a857894a8fa7d0b82
I20260812 03:06:36.589602 10622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: loading cluster ID for follower catalog manager: success
I20260812 03:06:36.615938 10622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: acquiring CA information for follower catalog manager: success
I20260812 03:06:36.665199 10651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 987a845b13194a52b77a487eddffd513: Generated new TSK 0
I20260812 03:06:36.666697 10651 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 03:06:36.669428 10622 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260812 03:06:36.875739 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:36.840603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.10.118.125:43283)
0812 03:06:36.856653 (+ 16050us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.856683 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.862382 (+ 5699us) server_negotiation.cc:407] Connection header received
0812 03:06:36.862567 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.862595 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.862819 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.863677 (+ 858us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.874580 (+ 10903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53858: BlockingRecv error: recv got EOF from 127.0.0.1:53858 (error 108)
Metrics: {"server-negotiator.queue_time_us":15007,"thread_start_us":15231,"threads_started":1}
W20260812 03:06:36.880915 10705 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:36.822690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.10.118.126:39375)
0812 03:06:36.857761 (+ 35071us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.857788 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.857846 (+ 58us) server_negotiation.cc:407] Connection header received
0812 03:06:36.858012 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.858037 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.858270 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.858498 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.864626 (+ 6128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44762: BlockingRecv error: recv got EOF from 127.0.0.1:44762 (error 108)
Metrics: {"server-negotiator.queue_time_us":34626,"thread_start_us":36096,"threads_started":1}
W20260812 03:06:36.893267 10711 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:36.866456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.10.118.124:46727)
0812 03:06:36.882405 (+ 15949us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.882428 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.882479 (+ 51us) server_negotiation.cc:407] Connection header received
0812 03:06:36.882626 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.882653 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.882887 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.883083 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.892080 (+ 8997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54222: BlockingRecv error: recv got EOF from 127.0.0.1:54222 (error 108)
Metrics: {"server-negotiator.queue_time_us":15484,"thread_start_us":15358,"threads_started":1}
W20260812 03:06:36.937101 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:36.928358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53862 (local address 127.10.118.125:43283)
0812 03:06:36.929000 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.929026 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.932665 (+ 3639us) server_negotiation.cc:407] Connection header received
0812 03:06:36.933663 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.933692 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.934004 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.934157 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.936126 (+ 1969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53862: BlockingRecv error: recv got EOF from 127.0.0.1:53862 (error 108)
Metrics: {"server-negotiator.queue_time_us":221}
W20260812 03:06:36.989144 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:36.976079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.10.118.125:43283)
0812 03:06:36.976655 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:36.976682 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:36.984780 (+ 8098us) server_negotiation.cc:407] Connection header received
0812 03:06:36.985738 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:36.985767 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:36.986052 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:36.986201 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:36.988111 (+ 1910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53864: BlockingRecv error: recv got EOF from 127.0.0.1:53864 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
W20260812 03:06:37.059666 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:37.041091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.10.118.125:43283)
0812 03:06:37.041601 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.041626 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.055660 (+ 14034us) server_negotiation.cc:407] Connection header received
0812 03:06:37.056279 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.056311 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.056644 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.056801 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:37.058722 (+ 1921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53874: BlockingRecv error: recv got EOF from 127.0.0.1:53874 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
I20260812 03:06:37.132071 10589 catalog_manager.cc:1259] Loaded cluster ID: 29da43d128b04c4a857894a8fa7d0b82
I20260812 03:06:37.132349 10589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: loading cluster ID for follower catalog manager: success
W20260812 03:06:37.134433 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:37.120800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.10.118.125:43283)
0812 03:06:37.121459 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.121494 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.129878 (+ 8384us) server_negotiation.cc:407] Connection header received
0812 03:06:37.130450 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.130479 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.130761 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.130907 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:37.133468 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53890: BlockingRecv error: recv got EOF from 127.0.0.1:53890 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 03:06:37.140228 10589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: acquiring CA information for follower catalog manager: success
I20260812 03:06:37.158497 10589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43f9dd776aef472ea9cd7c3ef56c9215: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 03:06:37.212742 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:37.188791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.10.118.125:43283)
0812 03:06:37.189325 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.189354 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.200257 (+ 10903us) server_negotiation.cc:407] Connection header received
0812 03:06:37.201221 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.201253 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.201535 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.201696 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:37.211729 (+ 10033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53896: BlockingRecv error: recv got EOF from 127.0.0.1:53896 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260812 03:06:37.298277 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:37.278250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.10.118.125:43283)
0812 03:06:37.278824 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.278851 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.293743 (+ 14892us) server_negotiation.cc:407] Connection header received
0812 03:06:37.294371 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.294421 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.294791 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.295029 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:37.297288 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53910: BlockingRecv error: recv got EOF from 127.0.0.1:53910 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
W20260812 03:06:37.384037 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:37.371987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.10.118.125:43283)
0812 03:06:37.372551 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.372577 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.379663 (+ 7086us) server_negotiation.cc:407] Connection header received
0812 03:06:37.380631 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.380663 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.380954 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.381115 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:37.383049 (+ 1934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53924: BlockingRecv error: recv got EOF from 127.0.0.1:53924 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
W20260812 03:06:37.517824 10708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:37.492215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.10.118.125:43283)
0812 03:06:37.492789 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.492817 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.513713 (+ 20896us) server_negotiation.cc:407] Connection header received
0812 03:06:37.514353 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.514381 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.514689 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.514847 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:37.516830 (+ 1983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53934: BlockingRecv error: recv got EOF from 127.0.0.1:53934 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260812 03:06:37.678217 10622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c45486c3ec1349d5a5e7d62b6bb3be1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 03:06:37.730938 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:37.703277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.10.118.125:43283)
0812 03:06:37.703902 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.703933 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.717868 (+ 13935us) server_negotiation.cc:407] Connection header received
0812 03:06:37.718591 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.718626 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.718975 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.719142 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:37.720818 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.722294 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.725415 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.726368 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.726604 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:37.727971 (+ 1367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:37.728668 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:37.728831 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:37.729282 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 03:06:37.729464 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260812 03:06:37.789475 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:37.771182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.10.118.124:46727)
0812 03:06:37.777845 (+ 6663us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.777867 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.777917 (+ 50us) server_negotiation.cc:407] Connection header received
0812 03:06:37.778109 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.778136 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.778484 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.778698 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:37.779815 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.781356 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.784048 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.785061 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.785341 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:37.786375 (+ 1034us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:37.787091 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:37.787292 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:37.787545 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 03:06:37.787746 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6247,"thread_start_us":5982,"threads_started":1}
I20260812 03:06:37.819926 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:37.804412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.10.118.126:39375)
0812 03:06:37.805762 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.805785 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.806996 (+ 1211us) server_negotiation.cc:407] Connection header received
0812 03:06:37.807635 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.807669 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.808019 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.808194 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:37.809616 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.811228 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.814147 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.815168 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.815417 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:37.816902 (+ 1485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:37.817425 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:37.817574 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:37.818033 (+ 459us) server_negotiation.cc:299] Negotiation successful
0812 03:06:37.818229 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":582,"threads_started":1}
I20260812 03:06:37.839434 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 10095
I20260812 03:06:37.841611 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:37.815916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54240 (local address 127.10.118.124:46727)
0812 03:06:37.816557 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.816584 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.821897 (+ 5313us) server_negotiation.cc:407] Connection header received
0812 03:06:37.823858 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.823898 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.824211 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.824400 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:37.825276 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.826686 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.833634 (+ 6948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.835757 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.836038 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:37.838674 (+ 2636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:37.838962 (+ 288us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:37.839102 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:37.839571 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 03:06:37.839786 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260812 03:06:37.847833 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:37.807180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.10.118.125:43283)
0812 03:06:37.807690 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:37.807720 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:37.818174 (+ 10454us) server_negotiation.cc:407] Connection header received
0812 03:06:37.818906 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:37.818939 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:37.819254 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:37.819389 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:37.822173 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.823541 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.832130 (+ 8589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:37.833099 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:37.833386 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:37.845219 (+ 11833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:37.845520 (+ 301us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:37.845675 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:37.846130 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 03:06:37.846348 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260812 03:06:37.887795 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 10237
I20260812 03:06:37.928150 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 10383
2026-08-12T03:06:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (10096 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 03:06:38 dist-test-slave-zpvx krb5kdc[10842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 03:06:38 dist-test-slave-zpvx krb5kdc[10842](info): set up 2 sockets
Aug 12 03:06:38 dist-test-slave-zpvx krb5kdc[10842](info): commencing operation
krb5kdc: starting...
W20260812 03:06:39.315228 10713 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/test-user.keytab.
Aug 12 03:06:39 dist-test-slave-zpvx krb5kdc[10842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T03:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T03:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.118.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
Entry for principal kudu/127.10.118.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab.
I20260812 03:06:39.553264 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36417
--webserver_interface=127.10.118.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:36417,127.10.118.125:41381,127.10.118.124:44587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 03:06:39.859232 10864 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 03:06:39.860134 10864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:39.860519 10864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:39.861177 10864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:39.891590 10864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:39.891959 10864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:39.892273 10864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:39.892549 10864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:39.892807 10864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:39.928735 10864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:36417,127.10.118.125:41381,127.10.118.124:44587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.126.keytab
--principal=kudu/127.10.118.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.126:36417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.118.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:39.930038 10864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:39.931107 10864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:39.948539 10872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:39.948522 10876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:39.948547 10879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:39.949643 10864 server_base.cc:1034] running on GCE node
I20260812 03:06:41.139367 10864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:41.142679 10864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:41.144129 10864 hybrid_clock.cc:648] HybridClock initialized: now 1786504001144037 us; error 105 us; skew 500 ppm
I20260812 03:06:41.145044 10864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:41 dist-test-slave-zpvx krb5kdc[10842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:41.156001 10864 init.cc:377] Logged in from keytab as kudu/127.10.118.126@KRBTEST.COM (short username kudu)
I20260812 03:06:41.161379 10864 webserver.cc:460] Webserver started at http://127.10.118.126:44855/ using document root <none> and password file <none>
I20260812 03:06:41.162282 10864 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:41.162477 10864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:41.162900 10864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:41.168192 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data/instance:
uuid: "001b7cac89de4cf8b07e738e1cde9590"
format_stamp: "Formatted at 2026-08-12 03:06:41 on dist-test-slave-zpvx"
server_key: "610935f4136883a14f809411c0c46570"
server_key_iv: "6cdfe491b8e282351ecc2d31a5997c26"
server_key_version: "encryptionkey@0"
I20260812 03:06:41.169430 10864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/wal/instance:
uuid: "001b7cac89de4cf8b07e738e1cde9590"
format_stamp: "Formatted at 2026-08-12 03:06:41 on dist-test-slave-zpvx"
server_key: "610935f4136883a14f809411c0c46570"
server_key_iv: "6cdfe491b8e282351ecc2d31a5997c26"
server_key_version: "encryptionkey@0"
I20260812 03:06:41.176910 10864 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:06:41.182890 10897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:41.183910 10864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260812 03:06:41.184245 10864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/wal
uuid: "001b7cac89de4cf8b07e738e1cde9590"
format_stamp: "Formatted at 2026-08-12 03:06:41 on dist-test-slave-zpvx"
server_key: "610935f4136883a14f809411c0c46570"
server_key_iv: "6cdfe491b8e282351ecc2d31a5997c26"
server_key_version: "encryptionkey@0"
I20260812 03:06:41.184660 10864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:41.228533 10864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:41.234107 10864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:41.234526 10864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:41.304713 10864 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.126:36417
I20260812 03:06:41.304943 10988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.126:36417 every 8 connection(s)
I20260812 03:06:41.307560 10864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/data/info.pb
I20260812 03:06:41.312695 10989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:41.316982 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 10864
I20260812 03:06:41.317440 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-0/wal/instance
I20260812 03:06:41.318514 10713 external_mini_cluster.cc:1428] Setting key 4b231fde3942a98b65aabe3beaee4f5a
I20260812 03:06:41.324267 10989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.118.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.125@KRBTEST.COM" created.
Aug 12 03:06:41 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:41.364822 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.328583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.10.118.126:36417)
0812 03:06:41.333619 (+ 5036us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:41.333652 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:41.333725 (+ 73us) server_negotiation.cc:407] Connection header received
0812 03:06:41.337324 (+ 3599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:41.337345 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:41.337611 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:41.337848 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:41.340078 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:41.341792 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:41.343890 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:41.344886 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:41.345235 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:41.351230 (+ 5995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:41.351256 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:41.351284 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:41.351386 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:41.356028 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:41.357362 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:41.357392 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:41.357407 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:41.357663 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:41.358421 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:41.358442 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:41.358457 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:41.359216 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:41.359546 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:41.360620 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0812 03:06:41.361417 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":232,"server-negotiator.queue_time_us":4552,"thread_start_us":3560,"threads_started":1}
I20260812 03:06:41.364874 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.328611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:36417 (local address 127.0.0.1:37770)
0812 03:06:41.331958 (+ 3347us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.332059 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:41.336877 (+ 4818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:41.337934 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:41.337961 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:41.339037 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:41.339753 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:41.339799 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:41.342083 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:41.342125 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:41.343638 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:41.343677 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:41.345294 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:41.345328 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:41.345938 (+ 610us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:41.346943 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:41.347046 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:41.350876 (+ 3830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:41.356279 (+ 5403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:41.356320 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:41.356354 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:41.357129 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:41.357882 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:41.357919 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:41.357936 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:41.358206 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:41.359468 (+ 1262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:41.359526 (+ 58us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:41.359939 (+ 413us) client_negotiation.cc:769] Sending connection context
0812 03:06:41.360297 (+ 358us) client_negotiation.cc:240] Negotiation successful
0812 03:06:41.361094 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2824,"mutex_wait_us":66,"spinlock_wait_cycles":19712,"thread_start_us":2345,"threads_started":1}
I20260812 03:06:41.368820 10989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
I20260812 03:06:41.371228 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.370087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38806)
0812 03:06:41.370611 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.370801 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260812 03:06:41.371749 10898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab.
Entry for principal kudu/127.10.118.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab.
I20260812 03:06:41.376288 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.374871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38818)
0812 03:06:41.375417 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.375660 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 03:06:41.376825 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:41381: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
I20260812 03:06:41.377897 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.125:41381
--webserver_interface=127.10.118.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab
--principal=kudu/127.10.118.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:36417,127.10.118.125:41381,127.10.118.124:44587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 03:06:41.426384 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } attempt: 1
I20260812 03:06:41.428783 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.427676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38820)
0812 03:06:41.428161 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.428328 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260812 03:06:41.430940 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.430023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38824)
0812 03:06:41.430443 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.430576 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260812 03:06:41.431332 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:41381: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
I20260812 03:06:41.499805 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } attempt: 2
I20260812 03:06:41.502658 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.501108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38826)
0812 03:06:41.501849 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.502074 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260812 03:06:41.505309 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.503916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38832)
0812 03:06:41.504498 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.504746 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260812 03:06:41.505810 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:41381: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
I20260812 03:06:41.597277 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } attempt: 3
I20260812 03:06:41.599895 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.598510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38838)
0812 03:06:41.599065 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.599294 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 03:06:41.602568 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.601346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38842)
0812 03:06:41.601820 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.602017 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260812 03:06:41.603008 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:41381: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
W20260812 03:06:41.687045 11006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 03:06:41.687878 11006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:41.688241 11006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:41.688879 11006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:41.718820 11006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:41.719175 11006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:41.719506 11006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:41.719767 11006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:41.720016 11006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:41.746459 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } attempt: 4
I20260812 03:06:41.749011 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.747837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38854)
0812 03:06:41.748316 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.748558 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 03:06:41.751384 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:41.750402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38870)
0812 03:06:41.750780 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:41.750934 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 03:06:41.751785 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:41381: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
I20260812 03:06:41.756839 11006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:36417,127.10.118.125:41381,127.10.118.124:44587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.125.keytab
--principal=kudu/127.10.118.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.125:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.118.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:41.758039 11006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:41.759068 11006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:41.775771 11022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:42.051220 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } attempt: 5
I20260812 03:06:42.054064 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:42.052876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38874)
0812 03:06:42.053428 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:42.053605 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 03:06:42.056540 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:42.055395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38876)
0812 03:06:42.055871 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:42.056039 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260812 03:06:42.057000 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:41381: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
I20260812 03:06:42.604471 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } attempt: 6
I20260812 03:06:42.608983 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:42.606061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38878)
0812 03:06:42.608178 (+ 2117us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:42.608419 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1692,"thread_start_us":1416,"threads_started":1}
I20260812 03:06:42.613201 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:42.612194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38894)
0812 03:06:42.612658 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:42.612821 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260812 03:06:42.613643 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.125:41381: Network error: Client connection negotiation failed: client connection to 127.10.118.125:41381: connect: Connection refused (error 111)
W20260812 03:06:41.775909 11023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:41.776798 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:41.778430 11006 server_base.cc:1034] running on GCE node
I20260812 03:06:42.969074 11006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:42.971727 11006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:42.973088 11006 hybrid_clock.cc:648] HybridClock initialized: now 1786504002973052 us; error 64 us; skew 500 ppm
I20260812 03:06:42.973984 11006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:42 dist-test-slave-zpvx krb5kdc[10842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504002, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:42.980175 11006 init.cc:377] Logged in from keytab as kudu/127.10.118.125@KRBTEST.COM (short username kudu)
I20260812 03:06:42.985694 11006 webserver.cc:460] Webserver started at http://127.10.118.125:46421/ using document root <none> and password file <none>
I20260812 03:06:42.986634 11006 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:42.986878 11006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:42.987357 11006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:42.992817 11006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data/instance:
uuid: "795755a7af124c10b2fea0fece65b950"
format_stamp: "Formatted at 2026-08-12 03:06:42 on dist-test-slave-zpvx"
server_key: "b405aa8814daec1389adfd08a66dd8da"
server_key_iv: "abf0a921d8029787a194781a8498d6c6"
server_key_version: "encryptionkey@0"
I20260812 03:06:42.994020 11006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/wal/instance:
uuid: "795755a7af124c10b2fea0fece65b950"
format_stamp: "Formatted at 2026-08-12 03:06:42 on dist-test-slave-zpvx"
server_key: "b405aa8814daec1389adfd08a66dd8da"
server_key_iv: "abf0a921d8029787a194781a8498d6c6"
server_key_version: "encryptionkey@0"
I20260812 03:06:43.001499 11006 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260812 03:06:43.007311 11044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:43.008358 11006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 03:06:43.008750 11006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/wal
uuid: "795755a7af124c10b2fea0fece65b950"
format_stamp: "Formatted at 2026-08-12 03:06:42 on dist-test-slave-zpvx"
server_key: "b405aa8814daec1389adfd08a66dd8da"
server_key_iv: "abf0a921d8029787a194781a8498d6c6"
server_key_version: "encryptionkey@0"
I20260812 03:06:43.009169 11006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:43.051555 11006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:43.057024 11006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:43.057510 11006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:43.125994 11006 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.125:41381
I20260812 03:06:43.126086 11150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.125:41381 every 8 connection(s)
I20260812 03:06:43.128827 11006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/data/info.pb
I20260812 03:06:43.134079 11151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:43.138396 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 11006
I20260812 03:06:43.138877 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-1/wal/instance
I20260812 03:06:43.139915 10713 external_mini_cluster.cc:1428] Setting key 9e2f80a23ef0c639a387d7228c47f2f0
I20260812 03:06:43.145853 11151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.118.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.118.124@KRBTEST.COM" created.
Aug 12 03:06:43 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504002, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:43.182454 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.149781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.10.118.126:36417)
0812 03:06:43.151757 (+ 1976us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:43.151779 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:43.152884 (+ 1105us) server_negotiation.cc:407] Connection header received
0812 03:06:43.157233 (+ 4349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:43.157277 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:43.157535 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:43.157741 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:43.160319 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.161233 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.164342 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.165112 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.165323 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:43.170484 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:43.170520 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:43.170557 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:43.170712 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:43.175813 (+ 5101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:43.177482 (+ 1669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:43.177501 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:43.177517 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:43.177708 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:43.178583 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:43.178602 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:43.178614 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:43.179098 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:43.179288 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:43.180739 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0812 03:06:43.181074 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1485,"thread_start_us":1153,"threads_started":1}
I20260812 03:06:43.184713 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.149923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:36417 (local address 127.0.0.1:37776)
0812 03:06:43.152605 (+ 2682us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.152699 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:43.156871 (+ 4172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:43.157926 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:43.157949 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:43.158780 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:43.159953 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:43.160031 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.161504 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.161539 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:43.164108 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:43.164153 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.165391 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.165412 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:43.165738 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:43.166686 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:43.166792 (+ 106us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:43.170095 (+ 3303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:43.176045 (+ 5950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:43.176084 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:43.176162 (+ 78us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:43.177278 (+ 1116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:43.177924 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:43.177960 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:43.177992 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:43.178381 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:43.179329 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:43.179379 (+ 50us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:43.180086 (+ 707us) client_negotiation.cc:769] Sending connection context
0812 03:06:43.180561 (+ 475us) client_negotiation.cc:240] Negotiation successful
0812 03:06:43.181360 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2056,"thread_start_us":1618,"threads_started":1}
I20260812 03:06:43.187114 11151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.118.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab.
Entry for principal kudu/127.10.118.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab.
Aug 12 03:06:43 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504002, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
I20260812 03:06:43.201679 10713 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
/tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.124:44587
--webserver_interface=127.10.118.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab
--principal=kudu/127.10.118.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.118.84:40553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.118.126:36417,127.10.118.125:41381,127.10.118.124:44587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 03:06:43.216182 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.188845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.10.118.125:41381)
0812 03:06:43.191567 (+ 2722us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:43.191587 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:43.191624 (+ 37us) server_negotiation.cc:407] Connection header received
0812 03:06:43.191769 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:43.191803 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:43.192040 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:43.192193 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:43.193811 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.195457 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.197315 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.198115 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.198359 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:43.203131 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:43.203182 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:43.203232 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:43.203393 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:43.209470 (+ 6077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:43.210585 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:43.210625 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:43.210654 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:43.210947 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:43.211824 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:43.211845 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:43.211858 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:43.212575 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:43.212804 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:43.213824 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0812 03:06:43.214417 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":201,"server-negotiator.queue_time_us":2373,"thread_start_us":1661,"threads_started":1}
I20260812 03:06:43.216933 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.188807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38896)
0812 03:06:43.189402 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.189454 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:43.189690 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:43.192309 (+ 2619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:43.192332 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:43.193012 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:43.193516 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:43.193552 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.195724 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.195763 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:43.196987 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:43.197028 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.198403 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.198424 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:43.198792 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:43.199750 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:43.199818 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:43.202727 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:43.209709 (+ 6982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:43.209733 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:43.209750 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:43.210342 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:43.211212 (+ 870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:43.211253 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:43.211282 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:43.211590 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:43.212791 (+ 1201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:43.212828 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:43.213221 (+ 393us) client_negotiation.cc:769] Sending connection context
0812 03:06:43.213537 (+ 316us) client_negotiation.cc:240] Negotiation successful
0812 03:06:43.214176 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260812 03:06:43.219812 11151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } has no permanent_uuid. Determining permanent_uuid...
I20260812 03:06:43.222043 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.221090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42636)
0812 03:06:43.221480 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.221657 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260812 03:06:43.222543 11045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.226768 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.225846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42652)
0812 03:06:43.226265 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.226400 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260812 03:06:43.227154 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.276786 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 1
I20260812 03:06:43.279121 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.278063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42658)
0812 03:06:43.278556 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.278716 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 03:06:43.281394 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.280452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42668)
0812 03:06:43.280860 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.281004 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 03:06:43.281805 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.350251 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 2
I20260812 03:06:43.352558 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.351466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42682)
0812 03:06:43.351943 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.352096 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 03:06:43.354851 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.353790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42684)
0812 03:06:43.354197 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.354337 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260812 03:06:43.355253 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.446731 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 3
I20260812 03:06:43.449127 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.447954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42694)
0812 03:06:43.448519 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.448681 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260812 03:06:43.451318 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.450348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42708)
0812 03:06:43.450771 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.450910 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260812 03:06:43.451686 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
W20260812 03:06:43.529198 11164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 03:06:43.530046 11164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 03:06:43.530337 11164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 03:06:43.530969 11164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 03:06:43.563400 11164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 03:06:43.563736 11164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 03:06:43.564034 11164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 03:06:43.564332 11164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 03:06:43.564647 11164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 03:06:43.595144 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 4
I20260812 03:06:43.597805 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.596478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42722)
0812 03:06:43.597031 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.597242 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 03:06:43.600500 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.599280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42738)
0812 03:06:43.599740 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.599945 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260812 03:06:43.600907 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.602177 11164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.118.84:40553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.118.126:36417,127.10.118.125:41381,127.10.118.124:44587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/krb5kdc/kudu_127.10.118.124.keytab
--principal=kudu/127.10.118.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.118.124:44587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.118.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 02:18:47 UTC on ccycloud.quasar-muerdu.root.comops.site
build id 4581826
TSAN enabled
I20260812 03:06:43.603426 11164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 03:06:43.604635 11164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 03:06:43.621745 11183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 03:06:43.674289 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } attempt: 7
Aug 12 03:06:43 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
I20260812 03:06:43.703984 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.676331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.10.118.125:41381)
0812 03:06:43.677026 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:43.677063 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:43.678719 (+ 1656us) server_negotiation.cc:407] Connection header received
0812 03:06:43.679246 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:43.679274 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:43.679470 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:43.679613 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:43.682442 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.683351 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.685270 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.685816 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.686094 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:43.691941 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:43.691962 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:43.691982 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:43.692080 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:43.695657 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:43.697592 (+ 1935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:43.697614 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:43.697628 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:43.697832 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:43.698997 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:43.699047 (+ 50us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:43.699083 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:43.700081 (+ 998us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:43.700491 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:43.701295 (+ 804us) server_negotiation.cc:299] Negotiation successful
0812 03:06:43.701750 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260812 03:06:43.704139 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.676044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38910)
0812 03:06:43.678467 (+ 2423us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.678543 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:43.678952 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:43.679798 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:43.679842 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:43.681297 (+ 1455us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:43.682071 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:43.682152 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.683658 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.683684 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:43.684999 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:43.685036 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:43.686096 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:43.686127 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:43.686493 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:43.687322 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:43.687384 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:43.691544 (+ 4160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:43.695900 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:43.695945 (+ 45us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:43.695985 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:43.697313 (+ 1328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:43.698127 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:43.698188 (+ 61us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:43.698225 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:43.698626 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:43.700348 (+ 1722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:43.700428 (+ 80us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:43.700790 (+ 362us) client_negotiation.cc:769] Sending connection context
0812 03:06:43.701060 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 03:06:43.701653 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1769,"thread_start_us":1357,"threads_started":1}
I20260812 03:06:43.706223 10989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } has no permanent_uuid. Determining permanent_uuid...
I20260812 03:06:43.708642 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.707635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42744)
0812 03:06:43.708052 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.708198 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 03:06:43.712687 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.711740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42760)
0812 03:06:43.712127 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.712271 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260812 03:06:43.713122 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.771515 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 1
I20260812 03:06:43.774128 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.773069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42770)
0812 03:06:43.773550 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.773721 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260812 03:06:43.776778 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.775759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42778)
0812 03:06:43.776205 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.776351 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260812 03:06:43.777254 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.858640 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 2
I20260812 03:06:43.861332 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.860214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42782)
0812 03:06:43.860748 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.860916 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260812 03:06:43.863924 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.862943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42786)
0812 03:06:43.863396 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.863543 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260812 03:06:43.864439 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.900395 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 5
I20260812 03:06:43.903885 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.901912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42788)
0812 03:06:43.902703 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.903011 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260812 03:06:43.907398 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.905585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42804)
0812 03:06:43.906266 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.906561 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260812 03:06:43.907953 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:43.949833 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 3
I20260812 03:06:43.952083 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.950982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42808)
0812 03:06:43.951484 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.951666 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260812 03:06:43.954645 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:43.953697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42810)
0812 03:06:43.954128 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:43.954270 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260812 03:06:43.955107 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:44.095721 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 4
I20260812 03:06:44.098408 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.097351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42818)
0812 03:06:44.097839 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.098001 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260812 03:06:44.101945 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.100139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42824)
0812 03:06:44.100837 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.101152 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260812 03:06:44.102494 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:44.385936 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 5
I20260812 03:06:44.389060 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.387038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42828)
0812 03:06:44.387833 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.388160 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260812 03:06:44.392419 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.390646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42832)
0812 03:06:44.391301 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.391596 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260812 03:06:44.392951 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:44.455403 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 6
I20260812 03:06:44.459331 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.456642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42848)
0812 03:06:44.458665 (+ 2023us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.458837 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":1404,"threads_started":1}
I20260812 03:06:44.463148 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.462161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42854)
0812 03:06:44.462615 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.462765 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260812 03:06:44.463617 11151 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
W20260812 03:06:43.622048 11176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:43.621747 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 03:06:44.819362 11180 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260812 03:06:44.819502 11164 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260812 03:06:44.823887 11164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 03:06:44.826622 11164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 03:06:44.828030 11164 hybrid_clock.cc:648] HybridClock initialized: now 1786504004828000 us; error 42 us; skew 500 ppm
I20260812 03:06:44.828926 11164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 03:06:44 dist-test-slave-zpvx krb5kdc[10842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 03:06:44.834861 11164 init.cc:377] Logged in from keytab as kudu/127.10.118.124@KRBTEST.COM (short username kudu)
I20260812 03:06:44.839488 11164 webserver.cc:460] Webserver started at http://127.10.118.124:35077/ using document root <none> and password file <none>
I20260812 03:06:44.840337 11164 fs_manager.cc:359] Metadata directory not provided
I20260812 03:06:44.840587 11164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 03:06:44.841023 11164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 03:06:44.846302 11164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data/instance:
uuid: "0256b91111984bc6a46f5d9c9fd78595"
format_stamp: "Formatted at 2026-08-12 03:06:44 on dist-test-slave-zpvx"
server_key: "3d887354fae87aafbcbdcc53915011a9"
server_key_iv: "a8a11ad73c5a45420303a6487e12cf0e"
server_key_version: "encryptionkey@0"
I20260812 03:06:44.847437 11164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/wal/instance:
uuid: "0256b91111984bc6a46f5d9c9fd78595"
format_stamp: "Formatted at 2026-08-12 03:06:44 on dist-test-slave-zpvx"
server_key: "3d887354fae87aafbcbdcc53915011a9"
server_key_iv: "a8a11ad73c5a45420303a6487e12cf0e"
server_key_version: "encryptionkey@0"
I20260812 03:06:44.854974 11164 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260812 03:06:44.860646 11212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 03:06:44.861781 11164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 03:06:44.862174 11164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data,/tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/wal
uuid: "0256b91111984bc6a46f5d9c9fd78595"
format_stamp: "Formatted at 2026-08-12 03:06:44 on dist-test-slave-zpvx"
server_key: "3d887354fae87aafbcbdcc53915011a9"
server_key_iv: "a8a11ad73c5a45420303a6487e12cf0e"
server_key_version: "encryptionkey@0"
I20260812 03:06:44.862524 11164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 03:06:44.902262 11164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 03:06:44.908291 11164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 03:06:44.908865 11164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 03:06:44.945389 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 6
I20260812 03:06:44.949551 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.946546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42864)
0812 03:06:44.948582 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.948764 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1624,"thread_start_us":1373,"threads_started":1}
I20260812 03:06:44.953326 11258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.952249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42874)
0812 03:06:44.952711 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:44.952864 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260812 03:06:44.953713 10989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.118.124:44587: Network error: Client connection negotiation failed: client connection to 127.10.118.124:44587: connect: Connection refused (error 111)
I20260812 03:06:44.983824 11164 rpc_server.cc:307] RPC server started. Bound to: 127.10.118.124:44587
I20260812 03:06:44.983925 11316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.118.124:44587 every 8 connection(s)
I20260812 03:06:44.986608 11164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/data/info.pb
I20260812 03:06:44.991894 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 03:06:44.994508 10713 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu as pid 11164
I20260812 03:06:44.995223 10713 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskx4bzWQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786503542438041-10713-0/minicluster-data/master-2/wal/instance
I20260812 03:06:44.996500 10713 external_mini_cluster.cc:1428] Setting key 17a2597ed0c250859697e679bb7a3b83
I20260812 03:06:45.010560 11317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 03:06:45 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
Aug 12 03:06:45 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
I20260812 03:06:45.036906 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:44.998800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.10.118.124:44587)
0812 03:06:45.001409 (+ 2609us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.001449 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.001530 (+ 81us) server_negotiation.cc:407] Connection header received
0812 03:06:45.006186 (+ 4656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.006249 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.006804 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.007259 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.009906 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.012504 (+ 2598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.014642 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.016017 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.016458 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.024094 (+ 7636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:45.024118 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:45.024144 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:45.024243 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:45.029649 (+ 5406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.031034 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.031092 (+ 58us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.031119 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.031600 (+ 481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.032257 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.032288 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.032302 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.033053 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:45.033510 (+ 457us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:45.034307 (+ 797us) server_negotiation.cc:299] Negotiation successful
0812 03:06:45.034822 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":1620,"threads_started":1}
I20260812 03:06:45.042640 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.015216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.10.118.126:36417)
0812 03:06:45.017295 (+ 2079us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.017316 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.018437 (+ 1121us) server_negotiation.cc:407] Connection header received
0812 03:06:45.018810 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.018828 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.019001 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.019136 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.021912 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.023136 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.025317 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.026286 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.026562 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.032199 (+ 5637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:45.032220 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:45.032242 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:45.032334 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:45.036575 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.037870 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.037890 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.037904 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.038090 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.038776 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.038795 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.038811 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.039399 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:45.039649 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:45.040622 (+ 973us) server_negotiation.cc:299] Negotiation successful
0812 03:06:45.041096 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1580,"thread_start_us":1186,"threads_started":1}
I20260812 03:06:45.043845 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.015139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.126:36417 (local address 127.0.0.1:37784)
0812 03:06:45.018187 (+ 3048us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:45.018287 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:45.018596 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:45.019315 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:45.019347 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:45.020714 (+ 1367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:45.021562 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.021616 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.023391 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.023421 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.025062 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.025099 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.026558 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.026578 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.026938 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.027784 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:45.027845 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:45.031886 (+ 4041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:45.036787 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.036820 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.036847 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.037647 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.038293 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.038323 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.038337 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.038564 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.039641 (+ 1077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:45.039674 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:45.039994 (+ 320us) client_negotiation.cc:769] Sending connection context
0812 03:06:45.040325 (+ 331us) client_negotiation.cc:240] Negotiation successful
0812 03:06:45.040944 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2550,"thread_start_us":1752,"threads_started":1}
I20260812 03:06:45.045929 11317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 03:06:45 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
Aug 12 03:06:45 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for kudu/127.10.118.125@KRBTEST.COM
I20260812 03:06:45.067797 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.040462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.10.118.125:41381)
0812 03:06:45.043768 (+ 3306us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.043782 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.043820 (+ 38us) server_negotiation.cc:407] Connection header received
0812 03:06:45.043921 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.043938 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.044193 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.044425 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.046569 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.047994 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.049883 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.050582 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.050855 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.055981 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:45.056015 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:45.056046 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:45.056180 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:45.061475 (+ 5295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.062665 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.062695 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.062719 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.062974 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.063606 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.063651 (+ 45us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.063674 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.064394 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:45.064849 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:45.065387 (+ 538us) server_negotiation.cc:299] Negotiation successful
0812 03:06:45.065697 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":2994,"thread_start_us":2749,"threads_started":1}
I20260812 03:06:45.070654 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.047328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.125:41381 (local address 127.0.0.1:38928)
0812 03:06:45.047901 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:45.047954 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:45.048200 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:45.050198 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:45.050224 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:45.050855 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:45.051138 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.051173 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.052521 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.052542 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.053554 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.053590 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.054653 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.054689 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.055044 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.055713 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:45.055825 (+ 112us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:45.058898 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:45.064849 (+ 5951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.064878 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.064903 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.065727 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.066443 (+ 716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.066463 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.066480 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.066680 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.067784 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:45.067801 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:45.068120 (+ 319us) client_negotiation.cc:769] Sending connection context
0812 03:06:45.068355 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 03:06:45.068792 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260812 03:06:45.071071 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.047368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.10.118.125:41381)
0812 03:06:45.049640 (+ 2272us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.049655 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.049688 (+ 33us) server_negotiation.cc:407] Connection header received
0812 03:06:45.049782 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.049798 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.049951 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.050131 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.051423 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.052261 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.053791 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.054389 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.054665 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.059210 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:45.059247 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:45.059280 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:45.059415 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:45.064616 (+ 5201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.065962 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.065990 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.066013 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.066244 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.066861 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.066892 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.066917 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.067580 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:45.067811 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:45.068657 (+ 846us) server_negotiation.cc:299] Negotiation successful
0812 03:06:45.069121 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"thread_start_us":1677,"threads_started":1}
I20260812 03:06:45.072924 11317 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 03:06:45 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786503999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.118.126@KRBTEST.COM
Aug 12 03:06:45 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504004, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.124@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
I20260812 03:06:45.095019 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.071451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.10.118.126:36417)
0812 03:06:45.071988 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.072016 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.072064 (+ 48us) server_negotiation.cc:407] Connection header received
0812 03:06:45.072197 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.072237 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.072503 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.072698 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.074918 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.076166 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.078354 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.079281 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.079547 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.085510 (+ 5963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:45.085544 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:45.085574 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:45.085709 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:45.089271 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.090661 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.090681 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.090695 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.090884 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.091495 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.091535 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.091550 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.092027 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:45.092482 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:45.093079 (+ 597us) server_negotiation.cc:299] Negotiation successful
0812 03:06:45.093533 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 03:06:45.096655 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.074068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:42896)
0812 03:06:45.074492 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:45.074522 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:45.074674 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:45.075393 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:45.075428 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:45.076136 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:45.076570 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.076604 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.078351 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.078381 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.080008 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.080072 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.081536 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.081557 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.081887 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.082582 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:45.082644 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:45.085792 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:45.090589 (+ 4797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.090613 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.090627 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.091184 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.091808 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.091841 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.091855 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.092050 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.093008 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:45.093036 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:45.093317 (+ 281us) client_negotiation.cc:769] Sending connection context
0812 03:06:45.093577 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 03:06:45.094108 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":65}
I20260812 03:06:45.096673 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.074236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.10.118.124:44587)
0812 03:06:45.074756 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.074781 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.074828 (+ 47us) server_negotiation.cc:407] Connection header received
0812 03:06:45.074953 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.074983 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.075141 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.075275 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.076807 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.078100 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.080331 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.081285 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.081567 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.086116 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:45.086141 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:45.086157 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:45.086250 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:45.090293 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.091384 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.091417 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.091435 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.091620 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.092250 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.092270 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.092283 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.092815 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:45.093059 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:45.093784 (+ 725us) server_negotiation.cc:299] Negotiation successful
0812 03:06:45.094385 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":135}
I20260812 03:06:45.102422 10713 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 03:06:45.105022 11317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: Bootstrap starting.
I20260812 03:06:45.110296 11317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:45.111980 11317 log.cc:824] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:45.117211 11317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: No bootstrap required, opened a new log
I20260812 03:06:45.114108 10946 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "001b7cac89de4cf8b07e738e1cde9590"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37800
I20260812 03:06:45.144308 11317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:45.145232 11317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0256b91111984bc6a46f5d9c9fd78595, State: Initialized, Role: FOLLOWER
I20260812 03:06:45.146051 11317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:45.149324 11332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:45.150180 11332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:45.151197 11317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 03:06:45.164770 11223 tablet.cc:2369] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 03:06:45.177188 11346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 03:06:45.177461 11346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 03:06:45.174542 11164 master_runner.cc:186] Error getting master registration for 127.10.118.126:36417: OK, instance_id { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" instance_seqno: 1786504001285671 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 03:06:45.208041 11164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 03:06:45.524256 11151 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 7
Aug 12 03:06:45 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504002, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.125@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
I20260812 03:06:45.552980 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.525884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.10.118.124:44587)
0812 03:06:45.526553 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.526582 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.527813 (+ 1231us) server_negotiation.cc:407] Connection header received
0812 03:06:45.528171 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.528196 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.528469 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.528681 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.530867 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.532250 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.534912 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.535861 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.536154 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.541747 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:45.541768 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:45.541793 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:45.541885 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:45.546595 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.547732 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.547760 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.547787 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.547983 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:45.548724 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:45.548757 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:45.548772 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:45.549271 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:45.549514 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:45.550293 (+ 779us) server_negotiation.cc:299] Negotiation successful
0812 03:06:45.550730 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 03:06:45.553293 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.525725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:48884)
0812 03:06:45.527594 (+ 1869us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:45.527655 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:45.527934 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:45.528808 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:45.528836 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:45.529969 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:45.530572 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.530621 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.532555 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.532585 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.534623 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.534685 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.536188 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.536221 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:45.536652 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:45.537626 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:45.537723 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:45.541418 (+ 3695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:45.546812 (+ 5394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.546842 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.546866 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.547526 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.548194 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:45.548225 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:45.548239 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:45.548520 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:45.549479 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:45.549510 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:45.549809 (+ 299us) client_negotiation.cc:769] Sending connection context
0812 03:06:45.550077 (+ 268us) client_negotiation.cc:240] Negotiation successful
0812 03:06:45.550626 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1438,"thread_start_us":1155,"threads_started":1}
I20260812 03:06:45.564188 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: Bootstrap starting.
I20260812 03:06:45.571290 11151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:45.573616 11151 log.cc:824] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:45.578959 11151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: No bootstrap required, opened a new log
I20260812 03:06:45.609556 11151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:45.610455 11151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795755a7af124c10b2fea0fece65b950, State: Initialized, Role: FOLLOWER
I20260812 03:06:45.611207 11151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:45.613772 11353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:45.614585 11353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:45.615747 11151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 03:06:45.643301 11367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 03:06:45.643678 11367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 03:06:45.639510 11006 master_runner.cc:186] Error getting master registration for 127.10.118.126:36417: OK, instance_id { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" instance_seqno: 1786504001285671 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 03:06:45.672552 11006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 03:06:45.814968 11055 tablet.cc:2369] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:06:45.987092 10989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } attempt: 7
Aug 12 03:06:46 dist-test-slave-zpvx krb5kdc[10842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786504001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.118.126@KRBTEST.COM for kudu/127.10.118.124@KRBTEST.COM
I20260812 03:06:46.031360 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.988558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.10.118.124:44587)
0812 03:06:45.989176 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:45.989206 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:45.990408 (+ 1202us) server_negotiation.cc:407] Connection header received
0812 03:06:45.991128 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:45.991157 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:45.991385 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:45.991541 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:45.995704 (+ 4163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.997125 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.000811 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.001754 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.001992 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:46.011285 (+ 9293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:46.011316 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:46.011342 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:46.011481 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:46.017917 (+ 6436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.020289 (+ 2372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.020347 (+ 58us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.020410 (+ 63us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.020853 (+ 443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.022241 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.022300 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.022332 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.023643 (+ 1311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:46.024018 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:46.026240 (+ 2222us) server_negotiation.cc:299] Negotiation successful
0812 03:06:46.027070 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260812 03:06:46.035497 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:45.988450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.118.124:44587 (local address 127.0.0.1:48900)
0812 03:06:45.990179 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0812 03:06:45.990254 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 03:06:45.990774 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 03:06:45.993166 (+ 2392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 03:06:45.993217 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 03:06:45.994573 (+ 1356us) client_negotiation.cc:189] Negotiated authn=SASL
0812 03:06:45.995278 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:45.995358 (+ 80us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:45.997553 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:45.997606 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:46.000405 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 03:06:46.000479 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.002681 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.002736 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 03:06:46.003449 (+ 713us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:46.004903 (+ 1454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 03:06:46.005045 (+ 142us) client_negotiation.cc:656] calling sasl_client_start()
0812 03:06:46.010782 (+ 5737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 03:06:46.018457 (+ 7675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:46.018512 (+ 55us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:46.018535 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:46.019921 (+ 1386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:46.021328 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 03:06:46.021379 (+ 51us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 03:06:46.021403 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 03:06:46.021873 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 03:06:46.024330 (+ 2457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 03:06:46.024361 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 03:06:46.025124 (+ 763us) client_negotiation.cc:769] Sending connection context
0812 03:06:46.032427 (+ 7303us) client_negotiation.cc:240] Negotiation successful
0812 03:06:46.033067 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"thread_start_us":705,"threads_started":1}
I20260812 03:06:46.045962 10989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590: Bootstrap starting.
I20260812 03:06:46.053325 10989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590: Neither blocks nor log segments found. Creating new log.
I20260812 03:06:46.055629 10989 log.cc:824] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590: Log is configured to *not* fsync() on all Append() calls
I20260812 03:06:46.066960 10989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590: No bootstrap required, opened a new log
I20260812 03:06:46.086926 10989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:46.087836 10989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b7cac89de4cf8b07e738e1cde9590, State: Initialized, Role: FOLLOWER
I20260812 03:06:46.088677 10989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:46.091384 11378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.092217 11378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:46.094853 10946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 03:06:46.095245 10946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:46.096508 10989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 03:06:46.101897 10946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:46.105682 11100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "001b7cac89de4cf8b07e738e1cde9590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "795755a7af124c10b2fea0fece65b950"
I20260812 03:06:46.106397 11100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:46.113394 11100 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b7cac89de4cf8b07e738e1cde9590 in term 1.
I20260812 03:06:46.114826 10898 leader_election.cc:302] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b7cac89de4cf8b07e738e1cde9590, 795755a7af124c10b2fea0fece65b950; no voters:
I20260812 03:06:46.115688 11378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 03:06:46.117709 11261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "001b7cac89de4cf8b07e738e1cde9590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0256b91111984bc6a46f5d9c9fd78595"
I20260812 03:06:46.118468 11261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 03:06:46.125574 11261 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b7cac89de4cf8b07e738e1cde9590 in term 1.
I20260812 03:06:46.127102 10946 leader_election.cc:288] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [CANDIDATE]: Term 1 election: Requested vote from peers 795755a7af124c10b2fea0fece65b950 (127.10.118.125:41381), 0256b91111984bc6a46f5d9c9fd78595 (127.10.118.124:44587)
I20260812 03:06:46.127868 10946 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:37800 (request call id 2) took 1014 ms. Trace:
I20260812 03:06:46.128321 10946 rpcz_store.cc:276] 0812 03:06:45.113690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0812 03:06:45.113834 (+ 144us) service_pool.cc:227] Handling call
0812 03:06:46.127809 (+1013975us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260812 03:06:46.130509 11378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [term 1 LEADER]: Becoming Leader. State: Replica: 001b7cac89de4cf8b07e738e1cde9590, State: Running, Role: LEADER
I20260812 03:06:46.131522 11378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } }
I20260812 03:06:46.153004 11384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001b7cac89de4cf8b07e738e1cde9590. Latest consensus state: current_term: 1 leader_uuid: "001b7cac89de4cf8b07e738e1cde9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.153939 11384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:46.168845 11391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260812 03:06:46.181710 11346 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 03:06:46.182045 11346 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 03:06:46.196815 11391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 03:06:46.212021 11414 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260812 03:06:46.244505 10909 tablet.cc:2369] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 03:06:46.253156 11261 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [term 1 FOLLOWER]: Refusing update from remote peer 001b7cac89de4cf8b07e738e1cde9590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:06:46.254972 11384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:06:46.264511 11100 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [term 1 FOLLOWER]: Refusing update from remote peer 001b7cac89de4cf8b07e738e1cde9590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 03:06:46.266208 11384 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 03:06:46.307504 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:46.204009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.10.118.125:41381)
0812 03:06:46.241593 (+ 37584us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.241619 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.253325 (+ 11706us) server_negotiation.cc:407] Connection header received
0812 03:06:46.253489 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.253517 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.253769 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.253942 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:46.261918 (+ 7976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.263423 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.266212 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.267134 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.267430 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:46.292694 (+ 25264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:46.292730 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:46.292761 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:46.292895 (+ 134us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:46.298611 (+ 5716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.300220 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.300253 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.300275 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.300535 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.301417 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.301459 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.301482 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.302173 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:46.302617 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:46.304686 (+ 2069us) server_negotiation.cc:299] Negotiation successful
0812 03:06:46.305151 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37166,"thread_start_us":37594,"threads_started":1}
I20260812 03:06:46.320976 11426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001b7cac89de4cf8b07e738e1cde9590. Latest consensus state: current_term: 1 leader_uuid: "001b7cac89de4cf8b07e738e1cde9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.321988 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:46.244784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.10.118.124:44587)
0812 03:06:46.245278 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.245302 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.256016 (+ 10714us) server_negotiation.cc:407] Connection header received
0812 03:06:46.256387 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.256416 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.256663 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.256854 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:46.265918 (+ 9064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.267254 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.280594 (+ 13340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.281705 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.281954 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:46.299894 (+ 17940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:46.299929 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:46.299959 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:46.300104 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:46.305477 (+ 5373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.307849 (+ 2372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.307878 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.307898 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.308137 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.312968 (+ 4831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.312998 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.313017 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.313611 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:46.313932 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:46.319473 (+ 5541us) server_negotiation.cc:299] Negotiation successful
0812 03:06:46.319935 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 03:06:46.321980 11426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:46.326457 11412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:46.190964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.10.118.126:36417)
0812 03:06:46.210954 (+ 19990us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.210981 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.243471 (+ 32490us) server_negotiation.cc:407] Connection header received
0812 03:06:46.252974 (+ 9503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.253004 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.253243 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.253404 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:46.259992 (+ 6588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.261797 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.270494 (+ 8697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.271468 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.271721 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:46.290074 (+ 18353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 03:06:46.290107 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 03:06:46.290135 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 03:06:46.290327 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0812 03:06:46.298999 (+ 8672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.308431 (+ 9432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.308461 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.308482 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.308747 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 03:06:46.309549 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 03:06:46.309579 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 03:06:46.309601 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0812 03:06:46.310267 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 03:06:46.310655 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:46.323472 (+ 12817us) server_negotiation.cc:299] Negotiation successful
0812 03:06:46.323962 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19550,"thread_start_us":23008,"threads_started":1}
I20260812 03:06:46.329355 11421 mvcc.cc:205] Tried to move back new op lower bound from 7317520409596395520 to 7317520409125617664. Current Snapshot: MvccSnapshot[applied={T|T < 7317520409596395520}]
I20260812 03:06:46.356164 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001b7cac89de4cf8b07e738e1cde9590. Latest consensus state: current_term: 1 leader_uuid: "001b7cac89de4cf8b07e738e1cde9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.357865 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:46.359051 11426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "001b7cac89de4cf8b07e738e1cde9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.359831 11426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:46.367483 11378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "001b7cac89de4cf8b07e738e1cde9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.368197 11378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:46.385862 11391 catalog_manager.cc:1347] Generated new cluster ID: 8d3ec4d537614c969cb5571435f11f9c
I20260812 03:06:46.386204 11391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 03:06:46.410239 11430 mvcc.cc:205] Tried to move back new op lower bound from 7317520409596395520 to 7317520409125617664. Current Snapshot: MvccSnapshot[applied={T|T < 7317520409596395520}]
I20260812 03:06:46.413115 11384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "001b7cac89de4cf8b07e738e1cde9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.413925 11384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590 [sys.catalog]: This master's current role is: LEADER
I20260812 03:06:46.441468 11425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "001b7cac89de4cf8b07e738e1cde9590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b7cac89de4cf8b07e738e1cde9590" member_type: VOTER last_known_addr { host: "127.10.118.126" port: 36417 } } peers { permanent_uuid: "795755a7af124c10b2fea0fece65b950" member_type: VOTER last_known_addr { host: "127.10.118.125" port: 41381 } } peers { permanent_uuid: "0256b91111984bc6a46f5d9c9fd78595" member_type: VOTER last_known_addr { host: "127.10.118.124" port: 44587 } } }
I20260812 03:06:46.442190 11425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 03:06:46.535898 11391 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 03:06:46.539072 11391 catalog_manager.cc:1504] Loading token signing keys...
I20260812 03:06:46.620042 11391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 001b7cac89de4cf8b07e738e1cde9590: Generated new TSK 0
I20260812 03:06:46.621403 11391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 03:06:46.650089 11367 catalog_manager.cc:1259] Loaded cluster ID: 8d3ec4d537614c969cb5571435f11f9c
I20260812 03:06:46.652464 11367 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: loading cluster ID for follower catalog manager: success
I20260812 03:06:46.672657 11367 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: acquiring CA information for follower catalog manager: success
I20260812 03:06:46.678936 11367 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 795755a7af124c10b2fea0fece65b950: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 03:06:46.780149 11412 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:46.762891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.10.118.126:36417)
0812 03:06:46.763426 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.763454 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.774712 (+ 11258us) server_negotiation.cc:407] Connection header received
0812 03:06:46.775907 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.775942 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.776214 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.776399 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:46.779010 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34146: BlockingRecv error: recv got EOF from 127.0.0.1:34146 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260812 03:06:46.788223 11419 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:46.770197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.10.118.125:41381)
0812 03:06:46.771232 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.771258 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.783138 (+ 11880us) server_negotiation.cc:407] Connection header received
0812 03:06:46.784145 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.784193 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.784591 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.784803 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:46.787055 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34406: BlockingRecv error: recv got EOF from 127.0.0.1:34406 (error 108)
Metrics: {"server-negotiator.queue_time_us":614}
W20260812 03:06:46.794061 11321 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:46.786942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.10.118.124:44587)
0812 03:06:46.787459 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.787484 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.789678 (+ 2194us) server_negotiation.cc:407] Connection header received
0812 03:06:46.790259 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.790291 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.790541 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.790710 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:46.792973 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48928: BlockingRecv error: recv got EOF from 127.0.0.1:48928 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260812 03:06:46.869411 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:46.851271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.10.118.125:41381)
0812 03:06:46.851884 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.851917 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.853783 (+ 1866us) server_negotiation.cc:407] Connection header received
0812 03:06:46.854479 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.854513 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.854897 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.855081 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:46.856838 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.858441 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.862989 (+ 4548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:46.864034 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:46.864307 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:46.865973 (+ 1666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:46.866768 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:46.866952 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:46.867429 (+ 477us) server_negotiation.cc:299] Negotiation successful
0812 03:06:46.867669 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
W20260812 03:06:46.934859 11321 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:46.915725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.10.118.124:44587)
0812 03:06:46.916314 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.916343 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:46.930439 (+ 14096us) server_negotiation.cc:407] Connection header received
0812 03:06:46.931155 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:46.931189 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:46.931536 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:46.931719 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:46.933831 (+ 2112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48936: BlockingRecv error: recv got EOF from 127.0.0.1:48936 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 03:06:47.028942 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:46.975713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.10.118.125:41381)
0812 03:06:46.976277 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:46.976307 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:47.003926 (+ 27619us) server_negotiation.cc:407] Connection header received
0812 03:06:47.015971 (+ 12045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:47.016005 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:47.016345 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:47.016515 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:47.018240 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.019810 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.023270 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.024277 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.024588 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:47.026101 (+ 1513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:47.026395 (+ 294us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:47.026519 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:47.027033 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 03:06:47.027240 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
W20260812 03:06:47.086639 11321 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 03:06:47.066922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.10.118.124:44587)
0812 03:06:47.067477 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:47.067506 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:47.080605 (+ 13099us) server_negotiation.cc:407] Connection header received
0812 03:06:47.082841 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:47.082877 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:47.083183 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:47.083365 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0812 03:06:47.085568 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48952: BlockingRecv error: recv got EOF from 127.0.0.1:48952 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260812 03:06:47.188185 11346 catalog_manager.cc:1259] Loaded cluster ID: 8d3ec4d537614c969cb5571435f11f9c
I20260812 03:06:47.188647 11346 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: loading cluster ID for follower catalog manager: success
I20260812 03:06:47.194435 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:47.144065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.10.118.125:41381)
0812 03:06:47.144656 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:47.144688 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:47.153756 (+ 9068us) server_negotiation.cc:407] Connection header received
0812 03:06:47.160614 (+ 6858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:47.160652 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:47.160976 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:47.161128 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:47.162655 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.164171 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.178696 (+ 14525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.179762 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.179998 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:47.181502 (+ 1504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:47.181827 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:47.181952 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:47.192592 (+ 10640us) server_negotiation.cc:299] Negotiation successful
0812 03:06:47.192816 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260812 03:06:47.214174 11346 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: acquiring CA information for follower catalog manager: success
I20260812 03:06:47.231453 11346 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0256b91111984bc6a46f5d9c9fd78595: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 03:06:47.281836 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:47.243656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.10.118.124:44587)
0812 03:06:47.244207 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:47.244233 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:47.260275 (+ 16042us) server_negotiation.cc:407] Connection header received
0812 03:06:47.268459 (+ 8184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:47.268491 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:47.268878 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:47.269072 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:47.270663 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.272119 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.275070 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.276071 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.276349 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:47.277918 (+ 1569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:47.278609 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:47.278777 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:47.279291 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 03:06:47.279494 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260812 03:06:47.369755 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:47.326924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34152 (local address 127.10.118.126:36417)
0812 03:06:47.353427 (+ 26503us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:47.353458 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:47.355352 (+ 1894us) server_negotiation.cc:407] Connection header received
0812 03:06:47.355597 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:47.355627 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:47.355996 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:47.356211 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:47.357816 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.359409 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.362967 (+ 3558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.364055 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.364345 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:47.365845 (+ 1500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:47.366392 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:47.366564 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:47.367638 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0812 03:06:47.367829 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26024,"thread_start_us":26452,"threads_started":1}
I20260812 03:06:47.379901 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 10864
I20260812 03:06:47.399046 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:47.344858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.10.118.125:41381)
0812 03:06:47.345420 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:47.345444 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:47.369915 (+ 24471us) server_negotiation.cc:407] Connection header received
0812 03:06:47.371508 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:47.371542 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:47.371842 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:47.371963 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:47.373458 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.374994 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.379064 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.380115 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.380413 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:47.396485 (+ 16072us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:47.396791 (+ 306us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:47.396946 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:47.397364 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 03:06:47.397575 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260812 03:06:47.417834 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 03:06:47.371528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.10.118.124:44587)
0812 03:06:47.373108 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0812 03:06:47.373133 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 03:06:47.398397 (+ 25264us) server_negotiation.cc:407] Connection header received
0812 03:06:47.399020 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 03:06:47.399056 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 03:06:47.399384 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 03:06:47.399522 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 03:06:47.408603 (+ 9081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.410014 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.412773 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 03:06:47.413688 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 03:06:47.413932 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 03:06:47.415226 (+ 1294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 03:06:47.415490 (+ 264us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 03:06:47.415649 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 03:06:47.416110 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 03:06:47.416298 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141}
I20260812 03:06:47.447345 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 11006
I20260812 03:06:47.483278 10713 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskx4bzWQ/build/tsan/bin/kudu with pid 11164
2026-08-12T03:06:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (9562 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (19659 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (464888 ms total)
[ PASSED ] 51 tests.
[ SKIPPED ] 5 tests, listed below:
[ SKIPPED ] SecurityITest.TestJwtMiniCluster
[ SKIPPED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
[ SKIPPED ] SecurityITest.IPKICACert
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
[ SKIPPED ] SecurityITest.TestEnableAndDisableEncryptedTSK
==================
WARNING: ThreadSanitizer: data race (pid=10713)
Write of size 8 at 0x7b0800003588 by main thread:
#0 operator delete(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 (security-itest+0x3abf6f)
#1 std::__1::_DeallocateCaller::__do_call(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:334:12 (security-itest+0x3da899)
#2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:292:12 (security-itest+0x3da839)
#3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:268:14 (libkrpc.so+0x1260d2)
#4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:340:3 (libkrpc.so+0x126069)
#5 std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::deallocate(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1798:10 (libkrpc.so+0x139d59)
#6 std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::deallocate(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1533:14 (libkrpc.so+0x139c69)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::destroy(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1837:9 (libkrpc.so+0x139b51)
#8 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~__tree() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1824:3 (libkrpc.so+0x139ac4)
#9 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~set() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:605:5 (libkrpc.so+0x132ea9)
#10 cxa_at_exit_wrapper(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:394:31 (security-itest+0x308f9f)
Previous read of size 8 at 0x7b0800003588 by thread T4:
#0 std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>* std::__1::__tree_next_iter<std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>*, std::__1::__tree_node_base<void*>*>(std::__1::__tree_node_base<void*>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:182:14 (libkrpc.so+0x13d84a)
#1 std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>::operator++() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:929:11 (libkrpc.so+0x133c62)
#2 void std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__assign_multi<std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> >(std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>, std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1696:31 (libkrpc.so+0x13b9c2)
#3 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1648:9 (libkrpc.so+0x13b819)
#4 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:540:21 (libkrpc.so+0x133a80)
#5 kudu::rpc::ClientNegotiation::SendNegotiate() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:327:20 (libkrpc.so+0x12b1c6)
#6 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:185:5 (libkrpc.so+0x12aa2b)
#7 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime, std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:231:3 (libkrpc.so+0x18081b)
#8 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:317:9 (libkrpc.so+0x17f962)
#9 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x19ed3c)
#10 decltype(std::__1::forward<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19ecb9)
#11 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19ec49)
#12 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19ec11)
#13 std::__1::__function::__func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19deed)
#14 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#15 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#16 kudu::ThreadPool::DispatchThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:776:7 (libkudu_util.so+0x45b6b6)
#17 kudu::ThreadPool::CreateThread()::$_2::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849:48 (libkudu_util.so+0x45eb11)
#18 decltype(std::__1::forward<kudu::ThreadPool::CreateThread()::$_2&>(fp)()) std::__1::__invoke<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x45eac9)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x45ea59)
#20 std::__1::__function::__alloc_func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x45ea21)
#21 std::__1::__function::__func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x45dd1d)
#22 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#23 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#24 kudu::Thread::SuperviseThread(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x43ddc6)
As if synchronized via sleep:
#0 nanosleep /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:363:3 (security-itest+0x334c6b)
#1 base::SleepForNanoseconds(long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/sysinfo.cc:96:10 (libgutil.so+0xb5eca)
#2 kudu::SleepFor(kudu::MonoDelta const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/monotime.cc:264:3 (libkudu_util.so+0x3d1516)
#3 kudu::Subprocess::KillAndWait(int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:675:7 (libkudu_util.so+0x430150)
#4 kudu::clock::MiniChronyd::Stop() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/clock/test/mini_chronyd.cc:242:16 (libmini_chronyd.so+0x150a0)
#5 kudu::clock::MiniChronyd::~MiniChronyd() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/clock/test/mini_chronyd.cc:159:5 (libmini_chronyd.so+0x14d9e)
#6 std::__1::default_delete<kudu::clock::MiniChronyd>::operator()(kudu::clock::MiniChronyd*) const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (libmini_cluster.so+0xc010e)
#7 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::reset(kudu::clock::MiniChronyd*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (libmini_cluster.so+0xc007d)
#8 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::~unique_ptr() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (libmini_cluster.so+0x9b6eb)
#9 std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >::destroy(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1811:92 (libmini_cluster.so+0xb0979)
#10 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1703:21 (libmini_cluster.so+0xb0950)
#11 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1544:14 (libmini_cluster.so+0xb0910)
#12 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destruct_at_end(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:428:9 (libmini_cluster.so+0xb08c6)
#13 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::clear() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:371:29 (libmini_cluster.so+0xb07e4)
#14 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~__vector_base() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:465:9 (libmini_cluster.so+0xb05db)
#15 std::__1::vector<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~vector() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:557:5 (libmini_cluster.so+0x96c11)
#16 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/mini-cluster/external_mini_cluster.cc:169:1 (libmini_cluster.so+0x8351e)
#17 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/mini-cluster/external_mini_cluster.cc:167:45 (libmini_cluster.so+0x83629)
#18 std::__1::default_delete<kudu::cluster::ExternalMiniCluster>::operator()(kudu::cluster::ExternalMiniCluster*) const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (security-itest+0x3e1f87)
#19 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::reset(kudu::cluster::ExternalMiniCluster*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (security-itest+0x3e1eed)
#20 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::~unique_ptr() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (security-itest+0x3e072b)
#21 kudu::SecurityITest::~SecurityITest() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:116:7 (security-itest+0x3d8247)
#22 kudu::ConnectToFollowerMasterTest::~ConnectToFollowerMasterTest() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1462:7 (security-itest+0x40d395)
#23 kudu::ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test::~ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1476:1 (security-itest+0x3d8759)
#24 testing::Test::DeleteSelf_() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:318:24 (libgtest.so.1.12.1+0x654c7)
#25 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e8f)
#26 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e8f)
#27 testing::TestInfo::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2859:5 (libgtest.so.1.12.1+0x43e22)
#28 testing::TestSuite::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d84)
#29 testing::internal::UnitTestImpl::RunAllTests() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59874)
#30 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65d4f)
#31 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65d4f)
#32 testing::UnitTest::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58e2c)
#33 RUN_ALL_TESTS() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (security-itest+0x42693b)
#34 main /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109:10 (security-itest+0x42583c)
Thread T4 'client-negotiat' (tid=11533, finished) created by thread T7 at:
#0 pthread_create /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966:3 (security-itest+0x30d042)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:638:15 (libkudu_util.so+0x43d62a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:146:12 (libmaster.so+0x2d3a39)
#3 kudu::ThreadPool::CreateThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:848:10 (libkudu_util.so+0x45a8bb)
#4 kudu::ThreadPool::DoSubmit(std::__1::function<void ()>, kudu::ThreadPoolToken*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:643:21 (libkudu_util.so+0x458cbf)
#5 kudu::ThreadPool::Submit(std::__1::function<void ()>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:525:10 (libkudu_util.so+0x45ae22)
#6 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x198c48)
#7 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr<kudu::rpc::Connection>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:591:14 (libkrpc.so+0x1996a1)
#8 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr<kudu::rpc::OutboundCall>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:375:14 (libkrpc.so+0x199266)
#9 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:907:14 (libkrpc.so+0x1a4bae)
#10 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:349:10 (libkrpc.so+0x1985d2)
#11 void ev::base<ev_async, ev::async>::method_thunk<kudu::rpc::ReactorThread, &(kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int))>(ev_loop*, ev_async*, int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0x1a5dd3)
#12 ev_invoke_pending /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x99b2)
#13 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:205:3 (libkrpc.so+0x1974a5)
#14 ev_run /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xab5c)
#15 ev::loop_ref::run(int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0x1a25b8)
#16 kudu::rpc::ReactorThread::RunThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510:9 (libkrpc.so+0x19aa85)
#17 kudu::rpc::ReactorThread::Init()::$_0::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197:48 (libkrpc.so+0x19d7d1)
#18 decltype(std::__1::forward<kudu::rpc::ReactorThread::Init()::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19d789)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19d719)
#20 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19d6e1)
#21 std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19c9dd)
#22 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#23 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#24 kudu::Thread::SuperviseThread(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x43ddc6)
SUMMARY: ThreadSanitizer: data race /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 in operator delete(void*)
==================
==================
WARNING: ThreadSanitizer: data race (pid=10713)
Write of size 8 at 0x7b0800003590 by main thread:
#0 operator delete(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 (security-itest+0x3abf6f)
#1 std::__1::_DeallocateCaller::__do_call(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:334:12 (security-itest+0x3da899)
#2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:292:12 (security-itest+0x3da839)
#3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:268:14 (libkrpc.so+0x1260d2)
#4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:340:3 (libkrpc.so+0x126069)
#5 std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::deallocate(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1798:10 (libkrpc.so+0x139d59)
#6 std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::deallocate(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1533:14 (libkrpc.so+0x139c69)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::destroy(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1837:9 (libkrpc.so+0x139b51)
#8 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~__tree() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1824:3 (libkrpc.so+0x139ac4)
#9 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~set() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:605:5 (libkrpc.so+0x132ea9)
#10 cxa_at_exit_wrapper(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:394:31 (security-itest+0x308f9f)
Previous read of size 8 at 0x7b0800003590 by thread T4:
#0 bool std::__1::__tree_is_left_child<std::__1::__tree_node_base<void*>*>(std::__1::__tree_node_base<void*>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:81:24 (libkrpc.so+0x13c38b)
#1 std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>* std::__1::__tree_next_iter<std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>*, std::__1::__tree_node_base<void*>*>(std::__1::__tree_node_base<void*>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:184:13 (libkrpc.so+0x13d874)
#2 std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>::operator++() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:929:11 (libkrpc.so+0x133c62)
#3 void std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__assign_multi<std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> >(std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>, std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1696:31 (libkrpc.so+0x13b9c2)
#4 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1648:9 (libkrpc.so+0x13b819)
#5 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:540:21 (libkrpc.so+0x133a80)
#6 kudu::rpc::ClientNegotiation::SendNegotiate() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:327:20 (libkrpc.so+0x12b1c6)
#7 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:185:5 (libkrpc.so+0x12aa2b)
#8 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime, std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:231:3 (libkrpc.so+0x18081b)
#9 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:317:9 (libkrpc.so+0x17f962)
#10 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x19ed3c)
#11 decltype(std::__1::forward<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19ecb9)
#12 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19ec49)
#13 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19ec11)
#14 std::__1::__function::__func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19deed)
#15 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#16 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#17 kudu::ThreadPool::DispatchThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:776:7 (libkudu_util.so+0x45b6b6)
#18 kudu::ThreadPool::CreateThread()::$_2::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849:48 (libkudu_util.so+0x45eb11)
#19 decltype(std::__1::forward<kudu::ThreadPool::CreateThread()::$_2&>(fp)()) std::__1::__invoke<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x45eac9)
#20 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x45ea59)
#21 std::__1::__function::__alloc_func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x45ea21)
#22 std::__1::__function::__func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x45dd1d)
#23 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#24 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#25 kudu::Thread::SuperviseThread(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x43ddc6)
As if synchronized via sleep:
#0 nanosleep /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:363:3 (security-itest+0x334c6b)
#1 base::SleepForNanoseconds(long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/sysinfo.cc:96:10 (libgutil.so+0xb5eca)
#2 kudu::SleepFor(kudu::MonoDelta const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/monotime.cc:264:3 (libkudu_util.so+0x3d1516)
#3 kudu::Subprocess::KillAndWait(int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:675:7 (libkudu_util.so+0x430150)
#4 kudu::clock::MiniChronyd::Stop() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/clock/test/mini_chronyd.cc:242:16 (libmini_chronyd.so+0x150a0)
#5 kudu::clock::MiniChronyd::~MiniChronyd() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/clock/test/mini_chronyd.cc:159:5 (libmini_chronyd.so+0x14d9e)
#6 std::__1::default_delete<kudu::clock::MiniChronyd>::operator()(kudu::clock::MiniChronyd*) const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (libmini_cluster.so+0xc010e)
#7 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::reset(kudu::clock::MiniChronyd*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (libmini_cluster.so+0xc007d)
#8 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::~unique_ptr() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (libmini_cluster.so+0x9b6eb)
#9 std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >::destroy(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1811:92 (libmini_cluster.so+0xb0979)
#10 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1703:21 (libmini_cluster.so+0xb0950)
#11 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1544:14 (libmini_cluster.so+0xb0910)
#12 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destruct_at_end(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:428:9 (libmini_cluster.so+0xb08c6)
#13 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::clear() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:371:29 (libmini_cluster.so+0xb07e4)
#14 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~__vector_base() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:465:9 (libmini_cluster.so+0xb05db)
#15 std::__1::vector<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~vector() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:557:5 (libmini_cluster.so+0x96c11)
#16 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/mini-cluster/external_mini_cluster.cc:169:1 (libmini_cluster.so+0x8351e)
#17 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/mini-cluster/external_mini_cluster.cc:167:45 (libmini_cluster.so+0x83629)
#18 std::__1::default_delete<kudu::cluster::ExternalMiniCluster>::operator()(kudu::cluster::ExternalMiniCluster*) const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (security-itest+0x3e1f87)
#19 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::reset(kudu::cluster::ExternalMiniCluster*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (security-itest+0x3e1eed)
#20 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::~unique_ptr() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (security-itest+0x3e072b)
#21 kudu::SecurityITest::~SecurityITest() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:116:7 (security-itest+0x3d8247)
#22 kudu::ConnectToFollowerMasterTest::~ConnectToFollowerMasterTest() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1462:7 (security-itest+0x40d395)
#23 kudu::ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test::~ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1476:1 (security-itest+0x3d8759)
#24 testing::Test::DeleteSelf_() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:318:24 (libgtest.so.1.12.1+0x654c7)
#25 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e8f)
#26 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e8f)
#27 testing::TestInfo::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2859:5 (libgtest.so.1.12.1+0x43e22)
#28 testing::TestSuite::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d84)
#29 testing::internal::UnitTestImpl::RunAllTests() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59874)
#30 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65d4f)
#31 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65d4f)
#32 testing::UnitTest::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58e2c)
#33 RUN_ALL_TESTS() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (security-itest+0x42693b)
#34 main /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109:10 (security-itest+0x42583c)
Thread T4 'client-negotiat' (tid=11533, finished) created by thread T7 at:
#0 pthread_create /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966:3 (security-itest+0x30d042)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:638:15 (libkudu_util.so+0x43d62a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:146:12 (libmaster.so+0x2d3a39)
#3 kudu::ThreadPool::CreateThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:848:10 (libkudu_util.so+0x45a8bb)
#4 kudu::ThreadPool::DoSubmit(std::__1::function<void ()>, kudu::ThreadPoolToken*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:643:21 (libkudu_util.so+0x458cbf)
#5 kudu::ThreadPool::Submit(std::__1::function<void ()>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:525:10 (libkudu_util.so+0x45ae22)
#6 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x198c48)
#7 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr<kudu::rpc::Connection>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:591:14 (libkrpc.so+0x1996a1)
#8 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr<kudu::rpc::OutboundCall>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:375:14 (libkrpc.so+0x199266)
#9 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:907:14 (libkrpc.so+0x1a4bae)
#10 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:349:10 (libkrpc.so+0x1985d2)
#11 void ev::base<ev_async, ev::async>::method_thunk<kudu::rpc::ReactorThread, &(kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int))>(ev_loop*, ev_async*, int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0x1a5dd3)
#12 ev_invoke_pending /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x99b2)
#13 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:205:3 (libkrpc.so+0x1974a5)
#14 ev_run /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xab5c)
#15 ev::loop_ref::run(int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0x1a25b8)
#16 kudu::rpc::ReactorThread::RunThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510:9 (libkrpc.so+0x19aa85)
#17 kudu::rpc::ReactorThread::Init()::$_0::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197:48 (libkrpc.so+0x19d7d1)
#18 decltype(std::__1::forward<kudu::rpc::ReactorThread::Init()::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19d789)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19d719)
#20 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19d6e1)
#21 std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19c9dd)
#22 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#23 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#24 kudu::Thread::SuperviseThread(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x43ddc6)
SUMMARY: ThreadSanitizer: data race /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 in operator delete(void*)
==================
==================
WARNING: ThreadSanitizer: data race (pid=10713)
Write of size 8 at 0x7b0800003598 by main thread:
#0 operator delete(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 (security-itest+0x3abf6f)
#1 std::__1::_DeallocateCaller::__do_call(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:334:12 (security-itest+0x3da899)
#2 std::__1::_DeallocateCaller::__do_deallocate_handle_size(void*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:292:12 (security-itest+0x3da839)
#3 std::__1::_DeallocateCaller::__do_deallocate_handle_size_align(void*, unsigned long, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:268:14 (libkrpc.so+0x1260d2)
#4 std::__1::__libcpp_deallocate(void*, unsigned long, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/new:340:3 (libkrpc.so+0x126069)
#5 std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::deallocate(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1798:10 (libkrpc.so+0x139d59)
#6 std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::deallocate(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, unsigned long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1533:14 (libkrpc.so+0x139c69)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::destroy(std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1837:9 (libkrpc.so+0x139b51)
#8 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~__tree() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1824:3 (libkrpc.so+0x139ac4)
#9 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::~set() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:605:5 (libkrpc.so+0x132ea9)
#10 cxa_at_exit_wrapper(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:394:31 (security-itest+0x308f9f)
Previous read of size 4 at 0x7b080000359c by thread T4:
#0 void std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >::construct<kudu::rpc::RpcFeatureFlag, kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag*, kudu::rpc::RpcFeatureFlag const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1809:35 (libkrpc.so+0x13cd9b)
#1 void std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::__construct<kudu::rpc::RpcFeatureFlag, kudu::rpc::RpcFeatureFlag const&>(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, kudu::rpc::RpcFeatureFlag*, kudu::rpc::RpcFeatureFlag const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1687:21 (libkrpc.so+0x13cd51)
#2 void std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > >::construct<kudu::rpc::RpcFeatureFlag, kudu::rpc::RpcFeatureFlag const&>(std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> >&, kudu::rpc::RpcFeatureFlag*, kudu::rpc::RpcFeatureFlag const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1538:14 (libkrpc.so+0x13cae1)
#3 std::__1::unique_ptr<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>, std::__1::__tree_node_destructor<std::__1::allocator<std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*> > > > std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__construct_node<kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:2194:5 (libkrpc.so+0x13c838)
#4 std::__1::__tree_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__emplace_multi<kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:2242:25 (libkrpc.so+0x13c6e6)
#5 std::__1::__tree_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__insert_multi<kudu::rpc::RpcFeatureFlag const&>(kudu::rpc::RpcFeatureFlag const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1326:16 (libkrpc.so+0x13bd58)
#6 void std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::__assign_multi<std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long> >(std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>, std::__1::__tree_const_iterator<kudu::rpc::RpcFeatureFlag, std::__1::__tree_node<kudu::rpc::RpcFeatureFlag, void*>*, long>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1697:9 (libkrpc.so+0x13b9ba)
#7 std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::__tree<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__tree:1648:9 (libkrpc.so+0x13b819)
#8 std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> >::operator=(std::__1::set<kudu::rpc::RpcFeatureFlag, std::__1::less<kudu::rpc::RpcFeatureFlag>, std::__1::allocator<kudu::rpc::RpcFeatureFlag> > const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/set:540:21 (libkrpc.so+0x133a80)
#9 kudu::rpc::ClientNegotiation::SendNegotiate() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:327:20 (libkrpc.so+0x12b1c6)
#10 kudu::rpc::ClientNegotiation::Negotiate(std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/client_negotiation.cc:185:5 (libkrpc.so+0x12aa2b)
#11 kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime, std::__1::unique_ptr<kudu::rpc::ErrorStatusPB, std::__1::default_delete<kudu::rpc::ErrorStatusPB> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:231:3 (libkrpc.so+0x18081b)
#12 kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, bool, kudu::MonoTime) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/negotiation.cc:317:9 (libkrpc.so+0x17f962)
#13 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x19ed3c)
#14 decltype(std::__1::forward<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19ecb9)
#15 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&>(kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19ec49)
#16 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19ec11)
#17 std::__1::__function::__func<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1, std::__1::allocator<kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&)::$_1>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19deed)
#18 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#19 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#20 kudu::ThreadPool::DispatchThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:776:7 (libkudu_util.so+0x45b6b6)
#21 kudu::ThreadPool::CreateThread()::$_2::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:849:48 (libkudu_util.so+0x45eb11)
#22 decltype(std::__1::forward<kudu::ThreadPool::CreateThread()::$_2&>(fp)()) std::__1::__invoke<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkudu_util.so+0x45eac9)
#23 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::ThreadPool::CreateThread()::$_2&>(kudu::ThreadPool::CreateThread()::$_2&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x45ea59)
#24 std::__1::__function::__alloc_func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x45ea21)
#25 std::__1::__function::__func<kudu::ThreadPool::CreateThread()::$_2, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_2>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x45dd1d)
#26 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#27 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#28 kudu::Thread::SuperviseThread(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x43ddc6)
As if synchronized via sleep:
#0 nanosleep /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:363:3 (security-itest+0x334c6b)
#1 base::SleepForNanoseconds(long) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/gutil/sysinfo.cc:96:10 (libgutil.so+0xb5eca)
#2 kudu::SleepFor(kudu::MonoDelta const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/monotime.cc:264:3 (libkudu_util.so+0x3d1516)
#3 kudu::Subprocess::KillAndWait(int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/subprocess.cc:675:7 (libkudu_util.so+0x430150)
#4 kudu::clock::MiniChronyd::Stop() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/clock/test/mini_chronyd.cc:242:16 (libmini_chronyd.so+0x150a0)
#5 kudu::clock::MiniChronyd::~MiniChronyd() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/clock/test/mini_chronyd.cc:159:5 (libmini_chronyd.so+0x14d9e)
#6 std::__1::default_delete<kudu::clock::MiniChronyd>::operator()(kudu::clock::MiniChronyd*) const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (libmini_cluster.so+0xc010e)
#7 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::reset(kudu::clock::MiniChronyd*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (libmini_cluster.so+0xc007d)
#8 std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >::~unique_ptr() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (libmini_cluster.so+0x9b6eb)
#9 std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >::destroy(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1811:92 (libmini_cluster.so+0xb0979)
#10 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1703:21 (libmini_cluster.so+0xb0950)
#11 void std::__1::allocator_traits<std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::destroy<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >(std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > >&, std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:1544:14 (libmini_cluster.so+0xb0910)
#12 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::__destruct_at_end(std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:428:9 (libmini_cluster.so+0xb08c6)
#13 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::clear() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:371:29 (libmini_cluster.so+0xb07e4)
#14 std::__1::__vector_base<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~__vector_base() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:465:9 (libmini_cluster.so+0xb05db)
#15 std::__1::vector<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> >, std::__1::allocator<std::__1::unique_ptr<kudu::clock::MiniChronyd, std::__1::default_delete<kudu::clock::MiniChronyd> > > >::~vector() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/vector:557:5 (libmini_cluster.so+0x96c11)
#16 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/mini-cluster/external_mini_cluster.cc:169:1 (libmini_cluster.so+0x8351e)
#17 kudu::cluster::ExternalMiniCluster::~ExternalMiniCluster() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/mini-cluster/external_mini_cluster.cc:167:45 (libmini_cluster.so+0x83629)
#18 std::__1::default_delete<kudu::cluster::ExternalMiniCluster>::operator()(kudu::cluster::ExternalMiniCluster*) const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2262:5 (security-itest+0x3e1f87)
#19 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::reset(kudu::cluster::ExternalMiniCluster*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2517:7 (security-itest+0x3e1eed)
#20 std::__1::unique_ptr<kudu::cluster::ExternalMiniCluster, std::__1::default_delete<kudu::cluster::ExternalMiniCluster> >::~unique_ptr() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/memory:2471:19 (security-itest+0x3e072b)
#21 kudu::SecurityITest::~SecurityITest() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:116:7 (security-itest+0x3d8247)
#22 kudu::ConnectToFollowerMasterTest::~ConnectToFollowerMasterTest() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1462:7 (security-itest+0x40d395)
#23 kudu::ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test::~ConnectToFollowerMasterTest_AuthnTokenVerifierHaveKeys_Test() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1476:1 (security-itest+0x3d8759)
#24 testing::Test::DeleteSelf_() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/include/gtest/gtest.h:318:24 (libgtest.so.1.12.1+0x654c7)
#25 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e8f)
#26 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e8f)
#27 testing::TestInfo::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2859:5 (libgtest.so.1.12.1+0x43e22)
#28 testing::TestSuite::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d84)
#29 testing::internal::UnitTestImpl::RunAllTests() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59874)
#30 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65d4f)
#31 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65d4f)
#32 testing::UnitTest::Run() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58e2c)
#33 RUN_ALL_TESTS() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (security-itest+0x42693b)
#34 main /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/test_main.cc:109:10 (security-itest+0x42583c)
Thread T4 'client-negotiat' (tid=11533, finished) created by thread T7 at:
#0 pthread_create /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966:3 (security-itest+0x30d042)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:638:15 (libkudu_util.so+0x43d62a)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.h:146:12 (libmaster.so+0x2d3a39)
#3 kudu::ThreadPool::CreateThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:848:10 (libkudu_util.so+0x45a8bb)
#4 kudu::ThreadPool::DoSubmit(std::__1::function<void ()>, kudu::ThreadPoolToken*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:643:21 (libkudu_util.so+0x458cbf)
#5 kudu::ThreadPool::Submit(std::__1::function<void ()>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/threadpool.cc:525:10 (libkudu_util.so+0x45ae22)
#6 kudu::rpc::ReactorThread::StartConnectionNegotiation(scoped_refptr<kudu::rpc::Connection> const&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:631:3 (libkrpc.so+0x198c48)
#7 kudu::rpc::ReactorThread::FindOrStartConnection(kudu::rpc::ConnectionId const&, kudu::rpc::CredentialsPolicy, scoped_refptr<kudu::rpc::Connection>*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:591:14 (libkrpc.so+0x1996a1)
#8 kudu::rpc::ReactorThread::AssignOutboundCall(std::__1::shared_ptr<kudu::rpc::OutboundCall>) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:375:14 (libkrpc.so+0x199266)
#9 kudu::rpc::AssignOutboundCallTask::Run(kudu::rpc::ReactorThread*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:907:14 (libkrpc.so+0x1a4bae)
#10 kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:349:10 (libkrpc.so+0x1985d2)
#11 void ev::base<ev_async, ev::async>::method_thunk<kudu::rpc::ReactorThread, &(kudu::rpc::ReactorThread::AsyncHandler(ev::async&, int))>(ev_loop*, ev_async*, int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:479:7 (libkrpc.so+0x1a5dd3)
#12 ev_invoke_pending /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3155:11 (libev.so.4+0x99b2)
#13 kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:205:3 (libkrpc.so+0x1974a5)
#14 ev_run /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/libev-4.20/ev.c:3555:7 (libev.so.4+0xab5c)
#15 ev::loop_ref::run(int) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/ev++.h:211:7 (libkrpc.so+0x1a25b8)
#16 kudu::rpc::ReactorThread::RunThread() /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:510:9 (libkrpc.so+0x19aa85)
#17 kudu::rpc::ReactorThread::Init()::$_0::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/reactor.cc:197:48 (libkrpc.so+0x19d7d1)
#18 decltype(std::__1::forward<kudu::rpc::ReactorThread::Init()::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899:1 (libkrpc.so+0x19d789)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x19d719)
#20 std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x19d6e1)
#21 std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator()() /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x19c9dd)
#22 std::__1::__function::__value_func<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libtserver_test_util.so+0x60204)
#23 std::__1::function<void ()>::operator()() const /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libtserver_test_util.so+0x60039)
#24 kudu::Thread::SuperviseThread(void*) /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x43ddc6)
SUMMARY: ThreadSanitizer: data race /data0/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 in operator delete(void*)
==================
ThreadSanitizer: reported 3 warnings